I have been having quite a time getting this to work reliably for 100s of thousands of terms and potentially millions of pages per source and ETL the resulting data into a database in an automated fashion. I need to run the tasks in Mesos on a repeating schedule. The required languages are Scala/Java.
However, the crawls constantly run into trouble in that they always hang despite being isolated fairly well and stripped down from one another (by specifying cache locations,minimizing the cache size, not downloading images;etc.).
Errors range from phantomjs returning a cleanup error and failing to continue to a general hang in Chrome Driver despite not running out of memory according to VisualVM. In fact, the highest memory use has been 25% and CPU use at 50% using 3-5 individual child processes.
Should I be running each term in a container? How to make web driver reliable over a period of weeks or months? Is there an equally generic alternative?