Workers and orchestrator

Question

I’m working on a system where we have several scheduled long running operations.

In our case this is website crawls that we perform for customers. The current setup is pragmatic where we have one service running that is responsible for distributing the work to four different “crawlers” that is running as threads in the same application.

All of this is coordinated using a queue in our database. Each “crawl” is scheduled to start at a given time and the orchestrator/coordinator fetches crawls from the db and assigns to a crawler. Each crawler reports is progress back to the coordinator in real-time which sends messages to other processes that wants to see the progress for each crawler. Each crawler also stores an internal state that is persisted so that it can resume after restarts etc.

I’m looking for ways to make this more scalable so that we could add more crawlers by spinning up new machines during high load etc.

We still want some kind of coordination so that we can get an overview of the system at any point in time “what if going on right now” but we want the number of crawlers so be more dynamic and scalable.

I’ve never had similar challenges but been thinking that each new machine could “check in for duty” in some way to make the coordinator aware and then start accepting jobs. I also feel that the over all pattern here must be something that has been “solved” before.

Does anyone have any pointers, ideas, or advice? Maybe there are frameworks or patterns that could help me in the right direction.

Our solution is built on a .NET6-stack.

Edit/Update: After evaluating some of the great answers here I realize that I need to add some more context to this. In a "straightforward" situation we could just have a pub/sub setup and have workers process messages on the queue. In our scenario, we need to keep track of the overall result of the crawl. For example, if website A has 150 URLs we need to crawl all of them and compare against the last time we did this. Since we're using WebDriver and a real browser instance we would save a lot of resources if the workers could "favor" continuing to process the same website and not jump between different sites. With an "orchestrator/coordinator" this would be quite easy since it would know which worker to assign for a given URL but I'm not sure how to accomplish the same thing with a broker.

On top of this, we also have rules that need to be applied before new URLs are added to the queue, things like exclusions etc. so there really needs to be something that keeps track of the overall progress, handles the exclusion etc. and delegates distinct tasks to the crawl-workers. In my mind, this also calls for some kind of orchestrator since putting this responsibility on each worker feels like it's doing to much.

Answer 1

I’m looking for ways to make this more scalable so that we could add more crawlers by spinning up new machines

It sounds like you already have a handle on this, and have implemented a sensible solution.

Coming at it from scratch, Kafka would be a reasonable way to satisfy the requirements, but there are lots of other pub-sub solutions such as 0mq.

A DB-centric approach could even work, where instead of event-driven scheduling the workers poll for tasks using SELECT queries. As long as you take care to index any timestamp columns, it should be straightforward to do "cheap" queries on URLs recently in need of attention.

Idle workers (beyond the 1st one) should drop out, so you're only billed for cloud compute that you actually need. A given number of backlogged crawl targets implies an estimated crawl completion time. Examine that statistic once per minute and keep spawning another worker until the estimate drops down to something acceptable to the business.

---

Here is a slightly sticky point for crawlers: time between probes matters. Hammering some poor webserver with a thousand GETs within a thousand milliseconds is antisocial.

Suppose you have half a dozen sites to crawl, with URL hostnames of A, B, ... , F. Each site has a thousand URLs. To be a good neighbor we insist on a delay of at least 2 seconds between GET requests, that is, 30 query/minute.

A naïve approach would ask a single worker to do all the A urls, then B, ... , finally the F urls. So we hammer A, or we spend a long time on sleep() to work through the A urls. Much better for that worker to sequentially visit the sites round-robin or to randomly shuffle the urls, so work on B .. F is perceived by A as "sleep" delay.

Now suppose we have two workers. How to coordinate? (with minimal communication!) Take SHA1 hash of each url and let worker-0 handle the even ones while worker-1 does the odd ones. With four workers we look at the two low-order hash bits, and in general mod N lets us field N workers.

We seldom will need to crawl the identical number of urls at each site, but the sketch above can still guide us. Peel off "large enough" batches of urls so that work done by crawlers and by origin web servers will be appropriately spread out, with no hotspots which impact the latency seen by production users. Order sites by number of urls, and be sure to incorporate the top few sites into the early batches.

Answer 2

As others mentioned, one can use a queue to distribute the workload. This response is different in that it distributes the workload by page granularity.

The queue here would be something like SQS where each message is brokered and not something like a (Kafka) stream. The queue is seeded by the scheduler with the website(s) to crawl. Crawlers (workers) listen for messages on the queue. (A worker instance can have multiple threads listening on the same queue.) Crawlers, as part of crawling the webpage, if they encounter links they need to crawl, they added a message on the same queue. After the page is crawled, the message is deleted from the queue.

This creates a robust architecture where workers can be added or removed as needed. Also if a worker goes down, the message reappears on the queue and is processed by another worker. A DB can be used to track the progress of the crawling and to track if a page was already crawled to avoid re-crawling.