I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple crawlers which produces data into Kafka.

My problem is that: when a crawler wants to crawl a website, other crawlers should not try to crawl it. How can I communicate them in a distributed environment?

1 Answer 1


If you already have Kafka, you could define another topic, that contains only URLs that need to be crawled.

Crawlers listen on this topic, poll the next URL to crawl. Crawlers should share a group id, so that each message is delivered only to a single crawler. When the crawlers detect new URLs, they don't crawl recursively, but put the new URLs into the URL topic.

You would have to create enough partitions for this topic (at least as many as there will be crawler instances, but should be more when possible).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.