Architecture solution for scalable crawling

Question

I have the following requirements:

1. a list of queries (rather long list ~ 100K - 1M items or even more) that I need to perform in a certain period of time and the execution time must be predictable. This process should be run daily

2. each query is performed against an external API

3. track progress of the entire process in the DB with showing % of completed queries

I would imagine a solution in which I have a producer which is run using CRON or similar that fetches the list from DB and pushes messages to the message broker and then to use the "competing consumers" pattern and workers to consume those messages and do the job. Then the question will be how to track the progress of the entire process and how to "know" that the worker finished processing. Can you let me know your thoughts on that solution and maybe there is something better than that. Thank you

Answer 1

I would suggest to starting with the simplest possible solution, and go to more complex solutions as needed. I would also recommend doing some estimation about the latency, CPU time, bandwith etc for a single request etc. This should help you estimate how complex solution and what hardware is needed.

Keeping to a single process will likely greatly simplify the implementation. While this will impose some limits, these limits can be fairly high with a powerful CPU and high bandwidth network card(s) etc. It is also possible that the process may bottleneck due to factors outside your control, like the speed of the target server(s), or the outgoing network connection. If that is the case, using multiple machines will likely have marginal improvement, and this can also put a cap on the scalability.

Note that 1M items is not really a lot, but it will greatly depend on how much work is done for each item. As a reference point, this article suggest the kestrel webserver can service ~18k requests per second on fairly modest hardware.

The simplest solution would just be a simple single threaded loop. The next step would be to make the loop parallel. However, using synchronous calls will be somewhat inefficient for IO work like network requests. To solve this you could use DataFlow with an asynchronous ActionBlock or TransformBlock with your desired degree of parallelism. Reporting progress could be done fairly simple by using interlocked.Increment on a shared field, and a timer that polls this field and updates the UI. Or use a Progress object to update the progress on the UI thread.

Moving to a solution capable on running on multiple machines would probably be similar to the DataFlow solution, but you would probably need to manage messages yourself using some message bus framework. I.e. have one component delegating work items to each worker, multiple workers on different machines, and one handling the result from each worker. The component handling the results should be able to show the progress as it receives the results. Running code on multiple machines also make deployment more complex, so you might want something like kubernetes to avoid the need for manual deployment.