Replacing in-memory queue with messaging queue service

Question

In our application (ASP.NET core) we have events that require computations and which results need to be persisted in a database (the objects are projects that need recalculations when a property changes, and then the financial results need to stored). These actions need to take place asynchronously in the background.

Right now we have implemented this with an in-memory queue and background service by implementing System.Threading.SemaphoreSlim and Microsoft.Extensions.Hosting.BackgroundService. What is great about this is that event is immediately triggered on queueing the workitem. What's not great about it is that the workitems are lost in the queue when the application is killed.

For a more robust solution we have migrated the queue to an Azure Storage Queue, in which we queue serialized workitem objects. The queue has worked well, and now we need to replace the BackgroundService.

After reviewing the Microsoft examples for de-queuing the message queue, I feel that polling directly makes most sense to me. This is because for example, Azure Queue storage trigger for Azure Functions also polls the queue, just with a simple algorithm. For the systems responsiveness a 5 second polling interval would meet our requirement. As such I have planned to implement Microsoft.Extensions.Hosting.IHostedService with a System.Threading.Timer for each queue (about a dozen queues) and poll my Azure storage queue every 5 seconds.

Q: Is there a more common or obvious implementation of a service that dequeues the Azure storage message queue in an ASP.NET core application? Or is this a sound approach to how one would use the Azure storage queue?

Answer 1

Assuming you've discounted a persistent queue hosted in whatever environment your processing app is, what you described seems like a reasonable plan to me. As for another more obvious implementation, you could consider moving your processing code to an Azure Function trigged by the queue entry. You gaining all the benefits of that (scale, reliability, processing pixie dust) but it's hard to know if it's viable or sensible without knowing the app. In any case, there's advantages in doing it in steps rather than big bang.

Depending on the work and complexity, your code-base etc. you might find a full port is a step too far. There's also the issue of "what else" does this product do? You can run the risk of fragmenting your code over a bunch of platforms which, while not impossible, comes with its own set of complexities (communication, configuration, security, documentation, trouble shooting, cost), advantages and disadvantages.