I’m working on a system where we have several scheduled long running operations.
In our case this is website crawls that we perform for customers. The current setup is pragmatic where we have one service running that is responsible for distributing the work to four different “crawlers” that is running as threads in the same application.
All of this is coordinated using a queue in our database. Each “crawl” is scheduled to start at a given time and the orchestrator/coordinator fetches crawls from the db and assigns to a crawler. Each crawler reports is progress back to the coordinator in real-time which sends messages to other processes that wants to see the progress for each crawler. Each crawler also stores an internal state that is persisted so that it can resume after restarts etc.
I’m looking for ways to make this more scalable so that we could add more crawlers by spinning up new machines during high load etc.
We still want some kind of coordination so that we can get an overview of the system at any point in time “what if going on right now” but we want the number of crawlers so be more dynamic and scalable.
I’ve never had similar challenges but been thinking that each new machine could “check in for duty” in some way to make the coordinator aware and then start accepting jobs. I also feel that the over all pattern here must be something that has been “solved” before.
Does anyone have any pointers, ideas, or advice? Maybe there are frameworks or patterns that could help me in the right direction.
Our solution is built on a .NET6-stack.
Edit/Update: After evaluating some of the great answers here I realize that I need to add some more context to this. In a "straightforward" situation we could just have a pub/sub setup and have workers process messages on the queue. In our scenario, we need to keep track of the overall result of the crawl. For example, if website A has 150 URLs we need to crawl all of them and compare against the last time we did this. Since we're using WebDriver and a real browser instance we would save a lot of resources if the workers could "favor" continuing to process the same website and not jump between different sites. With an "orchestrator/coordinator" this would be quite easy since it would know which worker to assign for a given URL but I'm not sure how to accomplish the same thing with a broker.
On top of this, we also have rules that need to be applied before new URLs are added to the queue, things like exclusions etc. so there really needs to be something that keeps track of the overall progress, handles the exclusion etc. and delegates distinct tasks to the crawl-workers. In my mind, this also calls for some kind of orchestrator since putting this responsibility on each worker feels like it's doing to much.