Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple crawlers which produces data into Kafka.
My problem is that: when a crawler wants to crawl a website, other crawlers should not try to crawl it. How can I communicate them in a distributed environment?
If you already have Kafka, you could define another topic, that contains only URLs that need to be crawled.
Crawlers listen on this topic, poll the next URL to crawl. Crawlers should share a group id, so that each message is delivered only to a single crawler. When the crawlers detect new URLs, they don't crawl recursively, but put the new URLs into the URL topic.
You would have to create enough partitions for this topic (at least as many as there will be crawler instances, but should be more when possible).
