Retry Error Handling for Activities and Orchestrations in Azure Durable Functions

Azure Durable Functions provides a rich set of Error Handling APIs. This post shows how Activities or Sub-Orchestrations can be re-run with the different retry options.

Activities in a workflow can call an API or run a code flow which might fail due to connection problems, network timeouts or other similar problems. If it was run a second time, it might succeed, and the flow could then complete successfully. For this, a retry Error Handling can be implemented. The HttpClient in .NET Core can implement this by using Polly. Azure Durable functions supports this directly without requiring extra Nuget packages. This can also be used to retry whole sub-orchestrations and not just single Http API calls.

Code: https://github.com/damienbod/AzureDurableFunctions

Posts in this series

Retrying Azure Durable Function Activities.

The IDurableOrchestrationContext interface provides the Retry Error Handling APIs in Azure Durable Functions. This interface is added as a parameter in the Azure Function using the OrchestrationTrigger.

[FunctionName(Constants.MyOrchestration)]
public async Task<MyOrchestrationDto> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{

The RetryOptions class provides the configuration for the different Retry calls. This can be used to set max timeouts, a fall off for retrying, or whatever configuration you might require.

The CallActivityWithRetryAsync method can then be called and if the Activity being called fails, the retry options will be used to re-run the activity until the max has be reached or it succeeds. If the activity continues to throw exceptions, the Durable function will complete with a Failed status, unless otherwise set.

var retryOptions = new RetryOptions(
	firstRetryInterval: TimeSpan.FromSeconds(3),
	maxNumberOfAttempts: 5)
{
	BackoffCoefficient = 1.5
};

var myActivityThreeResult = 
	await context.CallActivityWithRetryAsync<string> (
		Constants.MyActivityThree, 
		retryOptions, 
		context.GetInput<string>()
	);

Retrying Azure Durable Function Sub-Orchestrations.

Polly provides the same functionality when calling APIs. What’s nice about retry in Azure Durable Functions is that the Retry Error Handling can be applied to complete Orchestrations. If any part of the sub-orchestration fails, the whole thing can be re-run using this error handling. This can be implemented using the CallSubOrchestratorWithRetryAsync method.

var mySubOrchestrationDto = 
	await context.CallSubOrchestratorWithRetryAsync<MySubOrchestrationDto> (
		Constants.MySecondOrchestration, 
		retryOptions, 
		myActivityOne
	);

If we add an Exception one on the activities, we can see the retry APIs running and view the status of the orchestration running. (You need to set the showhistory=true)

Links