Increase service resilience using Polly and retry pattern in ASP.NET Core
Image from Pexels

Increase service resilience using Polly and retry pattern in ASP.NET Core

Retry pattern in ASP.NET Core with Polly

Whether you are doing micro-services or mid-size monolith applications that do not have to be too distributed, there is a big chance that you will depend on some external HTTP service. Whether it is REST, SOAP or any other type of the response, your application flow depends on it's response.

Sure using queues and event-based communication between services or components will definitely increase the resilience of the application and make more error-proof, it adds additional complexity to the whole solution and sometimes that can be an overhead depending on the application type or the infrastructure you might be limited with.

Another approach if you are depending on external services is to cache they response, but this is viable only in case you do not rely always on live data or the response from the services are lookups which are not updated frequently.

The only thing left is to retry your calls to the service that you depend on. You can find more about the retry pattern concept in Microsoft'd online documentation. This is not a complex thing to implement, but over time the complexity may grow and you eventually may end up with spaghetti code which is hard to maintain.

 Retry Pattern

source: https://docs.microsoft.com/en-us/azure/architecture/patterns/retry

An easy way is to use policies and there is no better library out there for policies than Polly. To make things even easier, Microsoft provides extension methods for using HTTP classes with Polly which makes building retry policy matter of just few lines which is easy to maintain as it allows you to configure policy in a fluent API manner. To try this out I made two projects inside a sample solution. One of the services will use the other one just to get passed text converted to uppercase and with letters in a reversed order. It will also check if the request is a retry or not by checking header value and depending on that it will either respond with proper result with status code 200 or with 404 not found HTTP status.

Sample unstable resource service

Let's first start with this sample resource service which will mimic failures. Since there is nothing special in the services and the request pipeline, I'll jump directly to the sample controller

using System;
using System.Linq;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace Experiments.Retry.Resource.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ValueController : ControllerBase
    {
       readonly ILogger<ValueController> _logger;

        public ValueController(ILogger<ValueController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public ActionResult<String> Get(String text)
        {
            //Try to get the retry header value
            var attempt = Request.Headers["X-Retry"];

            if (int.TryParse(attempt, out int attemptNo))
            {
                _logger.LogWarning($"RETRY: {attemptNo}");
                if (attemptNo < 3)
                {
                    //First or second retry
                    return NotFound();
                }
                else
                {
                    //Third retry
                    return Ok(new string(text.ToUpper().Reverse().ToArray()));
                }
            }

            //Not a retry - initial call
            return StatusCode(500);
        }
    }
}

    

The code is simple enough and it is obvious from the first look that

  • if there is no X-Retry header value in the request, method will respond with 500 status code,
  • in case X-Retry value in the headers is a number lower than 3, response will be 404 not found HTTP status code,
  • finally if X-Retry is 3 the proper response will be returned with 200 OK HTTP status code

Now that we have our fake unstable resource service let's focus how will we handle the failures on the invoker side. For the invoker application I will also use ASP.NET Core WebAPI project.

Sample invoker service

Before we start with any setup of logic, let add all necessary NuGet packages first

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.Extensions.Http" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Http.Polly" Version="2.2.0" />
  </ItemGroup>

</Project>

    

You may notice that we did not actually add Polly NuGet package. This is because we are referencing Microsoft.Extensions.Http.Polly that internally makes calls to Polly library (https://github.com/aspnet/Extensions/blob/master/src/HttpClientFactory/Polly/src/Microsoft.Extensions.Http.Polly.csproj).

The invoker service will be a bit more complex as we are going to put some setting in the IoC container setup in the Startup.cs. This will be initially HTTP client basic setup, so that we do not repeat ourself for every client.

As we may have multiple HTTP client setups, we can identify them by name which we can set in the when adding HttpClient setup to the IoC container in ConfigureService method in Startup.cs. In general using "magic" string in code may cause exceptions during the runtime, mostly because string value repetition or simply typos, I always first create static class where I list all my http client names for each client setup. So let's do that first:

using System;

namespace Experiments.Retry.Api
{
    public static class ServiceClientNames
    {
        public const String ResourceService = "resource-service";
    }
}

    

Now le's setup our client with some predefined values

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddHttpClient(ServiceClientNames.ResourceService, c =>
            {
                c.BaseAddress = new Uri("http://localhost:6000");
                c.DefaultRequestHeaders.Add("Accept", "application/json");
                c.DefaultRequestHeaders.Add("User-Agent", "Sample-Application");
            });

            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
        }
    

No magic here, we are still not ready to handle retries, but le't setup everything first without the retry pattern and see how it behaves. Next thing to be done is a controller which will call our unstable demo service which we created in the first place

using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;

namespace Experiments.Retry.Api.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class SampleController : ControllerBase
    {

        readonly IHttpClientFactory _httpClientFactory;

        public SampleController(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }

        [HttpGet]
        public async Task<ActionResult<String>> Get(String word)
        {
            if (!String.IsNullOrWhiteSpace(word))
            {
                var result = await _httpClientFactory
                    .CreateClient(ServiceClientNames.ResourceService)
                    .GetAsync($"/api/value?text={word}");
                if (result.IsSuccessStatusCode)
                {
                    return Ok(await result.Content.ReadAsStringAsync());
                }
            }
            return NoContent();
        }
    }
}

    

Nothing special here either. Let's run both of the services and try to invoke Sample controller method of the invoker service using POSTMAN.

Note

ASP.NET Core by default sets up HTTPS redirection in the pipeline. Although this is a great feature, it may cause problems with self signed certificate on localhost and POSTMAN. So, since this is just a demo solution, I commented out app.UseHttpsRedirection(); from the pipeline configuration (Configure) method in Startup.cs

Call Failed

Since subsequent call to a resource service failed, we responded with an empty result. Nothing happening after the request to resource service failed and this is where the whole request journey ends.

Now let's see how do we overcome this and retry for a resource service if we get 50x, 408 (timeout) or 404 http status. We need to do configuration in our IoC container to configure the IHttpClientFactory to return an instance of HttpClient with retry policy applied. Here is how ConfigureServices method will look like now:

        public void ConfigureServices(IServiceCollection services)
        {

            services.AddHttpClient(ServiceClientNames.ResourceService, c =>
            {
                c.BaseAddress = new Uri("http://localhost:6000");
                c.DefaultRequestHeaders.Add("Accept", "application/json");
                c.DefaultRequestHeaders.Add("User-Agent", "Sample-Application");
            })
                .AddPolicyHandler(p =>
                HttpPolicyExtensions
                    .HandleTransientHttpError()
                    .OrResult(msg => msg.StatusCode == HttpStatusCode.NotFound)
                    .Or<TimeoutRejectedException>()
                    .WaitAndRetryAsync(3, attempt =>
                    {

                        //Tell the service about attempt number
                        p.Headers.Remove("X-Retry");
                        p.Headers.Add("X-Retry", attempt.ToString());

                        //Set the wait interval
                        return TimeSpan.FromSeconds(attempt * 3);

                    })
                );

            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
        }
    

We setup the policy to retry for 3 times with dynamic interval which increases by 3 seconds after every failure. Now let's run the request and see what is happening.

 Call Retry

First thing that you immediately notice in the POSTMAN is that it takes a lot longer to get the response for the initiated request, but eventually you get the response with the proper value and 200 OK status code. As for the time, it is around 18 seconds which adds up to the calculation of the retry wait (1*3+2*3+3*3 seconds)

Whole solution with both projects is available from the public repository https://github.com/dejanstojanovic/Experiments.Retry

References

Disclaimer

Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.


About the author

DEJAN STOJANOVIC

Dejan is a passionate Software Architect/Developer. He is highly experienced in .NET programming platform including ASP.NET MVC and WebApi. He likes working on new technologies and exciting challenging projects

CONNECT WITH DEJAN  Loginlinkedin Logintwitter Logingoogleplus Logingoogleplus

JavaScript

read more

SQL/T-SQL

read more

Umbraco CMS

read more

PowerShell

read more

Comments for this article