I have the following requirements:

  1. a list of queries (rather long list ~ 100K - 1M items or even more) that I need to perform in a certain period of time and the execution time must be predictable. This process should be run daily

  2. each query is performed against an external API

  3. track progress of the entire process in the DB with showing % of completed queries

I would imagine a solution in which I have a producer which is run using CRON or similar that fetches the list from DB and pushes messages to the message broker and then to use the "competing consumers" pattern and workers to consume those messages and do the job. Then the question will be how to track the progress of the entire process and how to "know" that the worker finished processing. Can you let me know your thoughts on that solution and maybe there is something better than that. Thank you

  • I assume that each worker would post a message with the result of each job, and have some central component receiving these messages and report the overall progress.
    – JonasH
    Apr 8, 2022 at 12:18
  • “execution time must be predictable” – could you clarify what you mean by that? “how to ‘know’ that the worker finished processing” – how important is it that a query is retried if it fails or times out?
    – amon
    Apr 8, 2022 at 12:49
  • @amon “execution time must be predictable” - this means that I want to be able to scalable i.e. be able to increase the speed. For example I want those 100K queries to be performed in an hour or so. I know how much time each query takes approx so I can imagine adding more workers to increase the speed of crawling. I can also try using auto scaling by consuming some metric like queue length or similar Apr 8, 2022 at 13:26
  • @amon "how important is it that a query is retried if it fails or times out" - I am planning to use retries (maybe even by re-queuing messages?? to be able to control scaling using queue length) and probably a dead-letter queue to pick up those failed permanently Apr 8, 2022 at 13:29
  • @JonasH yeah that sounds a good solution but in that case should I have something like a stateful saga or something as a central component that knows how many are there to process and how many are already processed so that I know % completed? Apr 8, 2022 at 13:32

1 Answer 1


I would suggest to starting with the simplest possible solution, and go to more complex solutions as needed. I would also recommend doing some estimation about the latency, CPU time, bandwith etc for a single request etc. This should help you estimate how complex solution and what hardware is needed.

Keeping to a single process will likely greatly simplify the implementation. While this will impose some limits, these limits can be fairly high with a powerful CPU and high bandwidth network card(s) etc. It is also possible that the process may bottleneck due to factors outside your control, like the speed of the target server(s), or the outgoing network connection. If that is the case, using multiple machines will likely have marginal improvement, and this can also put a cap on the scalability.

Note that 1M items is not really a lot, but it will greatly depend on how much work is done for each item. As a reference point, this article suggest the kestrel webserver can service ~18k requests per second on fairly modest hardware.

The simplest solution would just be a simple single threaded loop. The next step would be to make the loop parallel. However, using synchronous calls will be somewhat inefficient for IO work like network requests. To solve this you could use DataFlow with an asynchronous ActionBlock or TransformBlock with your desired degree of parallelism. Reporting progress could be done fairly simple by using interlocked.Increment on a shared field, and a timer that polls this field and updates the UI. Or use a Progress object to update the progress on the UI thread.

Moving to a solution capable on running on multiple machines would probably be similar to the DataFlow solution, but you would probably need to manage messages yourself using some message bus framework. I.e. have one component delegating work items to each worker, multiple workers on different machines, and one handling the result from each worker. The component handling the results should be able to show the progress as it receives the results. Running code on multiple machines also make deployment more complex, so you might want something like kubernetes to avoid the need for manual deployment.

  • Thank you for your answer. We tried single threaded then multi threaded then multi process solution but now we got to the point where we potentially need to handle millions of those small tasks in a limited amount if time hence why we are thinking of moving it to multi process , multi node solution. The solution should be highly scalable so we are planning kubernetes cluster and that’s why we are considering a big architectural revision, plus it would be a great exercise to implement something like this. I have been given a great advice to learn some of the enterprise integration patterns Apr 25, 2022 at 20:53
  • @Yevgeniy Yemelin a multi-node solution might very well be the best solution for your needs. But I would still recommend doing some benchmarking and profiling so you have a good understanding of where the likely bottlenecks are. Just knowing how many tasks you have is meaningless unless you also know how much resource cpu/memory/bandwith etc each task needs.
    – JonasH
    Apr 26, 2022 at 9:38
  • thank you. I will consider using DataFlow when I get a chance to get familiar with it. My main concern about single node solution is it has theoretical limit for CPU/Memory/Connections etc and since we don't yet know the max amount of load we will need to process as the number of clients (and their demands) grow. Definitely worth learning DataFlow, thank you again! Apr 27, 2022 at 11:03
  • 2
    Keep in mind that you can't just keep throwing threads and nodes at a problem to get better performance. Eventually you're adding more over head then you're getting in improvements. Sep 16, 2022 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.