Orphaned Activity Function Stuck in Processing Loop

[brandonwatts]

🐛 Describe the bug

We are seeing activity functions that are stuck in a loop, calling every 5 minutes with no way to stop them. This occurred after we had a large batch of requests hit our service. Im assuming this caused some orchestrator/sub-orchestrator failures creating a bunch of orphaned activity functions. We at the very least want some way to locate & stop these orphaned functions as they are calling out to a REST api causing unneeded spend.

Its possibly its related to this issue from 5 years ago? It sounds very similar but could be a coincidence.

🤔 Expected behavior

What should have happened?
I would have expected these orphaned activity functions to be killed off after the sub-orchestrator failed.

☕ Steps to reproduce

We are using an orchestrator that calls a sub-orchestrator that then has a single activity function in which a REST call is made and returned. We are running the service in Azure via an App Service Plan.

We are using Python v2 durable functions. azure-functions-durable==1.2.9 & azure-functions==1.19.0. Version 3.0.0-rc.1 of DurableTask runtime.

⚡If deployed to Azure

• Timeframe issue observed: 8/2/24 - Current
• Function App name: prod-platds-content-func
• Function name(s): orchestrate_internal_links_request, suborchestrate_generate
• Azure region: South Central US
• Orchestration instance ID(s):
  • ac2707716f874d268af7083e2edef238:0
  • 14ae447dee7b447795a3a6dc0fb99ee3:0
  • 83ad90ef9c76477bb69bd8b98a51bb5e:0
  • ab43cf74566048bda45d9fcceaff6ddf:0)
  • I can provide more if needed.
• Azure storage account name: prodplatformdsscusst