I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple crawlers which produces data into Kafka.
My problem is that: when a crawler wants to crawl a website, other crawlers should not try to crawl it. How can I communicate them in a distributed environment?