4

I have been having quite a time getting this to work reliably for 100s of thousands of terms and potentially millions of pages per source and ETL the resulting data into a database in an automated fashion. I need to run the tasks in Mesos on a repeating schedule. The required languages are Scala/Java.

For acquisition, I need to parse javascript, render data from ajax, work with tracking cookies; etc. in order to scrape the sites. I've been working on an open source tool to do this as well. I discovered and have created an extremely simple API surrounding Selenium for this task with serializable configuration for distribution. The tool is plug and play for a webdriver.

However, the crawls constantly run into trouble in that they always hang despite being isolated fairly well and stripped down from one another (by specifying cache locations,minimizing the cache size, not downloading images;etc.).

Errors range from phantomjs returning a cleanup error and failing to continue to a general hang in Chrome Driver despite not running out of memory according to VisualVM. In fact, the highest memory use has been 25% and CPU use at 50% using 3-5 individual child processes.

Should I be running each term in a container? How to make web driver reliable over a period of weeks or months? Is there an equally generic alternative?

Improve this question
1
  • In response to the answer. I was using a jaunt style approach initially but, unfortunately, due to the perceived value of data, persistance of javascript, and growing ant-scraping field including captchas requiring RNNs/Convolutional neural nets to classify what is in them or speech recognition on par with Watson Speech Rec (guess where my .04 cents per minute goes), ip tracking, and variable setting, the old days of purely static calls are gone. Jaunt itself is pretty poor. Take a look at my own tool github.com/asevans48/JScrape for a more configurable approach which I will add JS to
    – user3916597
    Commented Oct 27, 2016 at 20:58

2 Answers 2

Reset to default
1

This may not be the most satisfactory type of answer, but the fact is that web browsers are not built and tested with the expectation that they will be run continuously for weeks or months, while fetching hundreds of thousands of pages.

While the browser developers of course do their best to make their software work reliably during long browsing sessions, problems that only appear in extreme use cases are unlikely to get the highest level of attention.

Therefore, first of all, try to recycle browser processes on a regular basis. After every few thousand pages fetched might be a good starting point.

Second most important, try different browser types. Perhaps Chrome/Chromium/PhantomJS is not currently the the most stable browser type for your use case. (At the time of writing, PhantomJS is no longer under active development.) A great advantage of using Selenium is that many different types of browsers are supported. Try Firefox and see if performance is better. Or if the platform supports it, try Edge or Safari.

Third, make sure that unexpected situations are handled in ways that do not consume too much resources. For example, if links to PDF files, downloadable documents, etc are encountered during scraping, or if popup windows, new tabs or dialog boxes are opened. Some websites link to print versions of pages that automatically open a print dialog.

Improve this answer
0

Most websites do not require that you stand up an entire browser and automation framework to scrape them; doing so marshals a lot of machinery that is simply not necessary, and introduces a lot of complexity that reduces your overall system reliability.

For examples on how to write scrapers that are more lightweight and more reliable, a look at Jaunt: http://jaunt-api.com/

Improve this answer
8
  • Lets just say, I am not as welcome on the sources I need to scrape, they use a javascript generated viewstate, contain loads of ajax, or have some sort of custom generating variable. Otherwise, I usually stick to Apache HTTP Components because it has a thread pool. I have found three levels of requirements based on this and the need to get loops. My 1st option is always Apache HTTP Components, next is to mix them because results require cookies but getting the cookies requires some sort of JS; etc. and the most dreaded (the staller), requires the works.
    – user3916597
    Commented Oct 19, 2016 at 20:51
  • I haven't had much luck doing this kind of scraping. All of the headless browsers are a bit wonky.
    – Robert Harvey
    Commented Oct 19, 2016 at 20:54
  • I used to use PyQt but it has just been dropping the ball lately. I keep having to kick off my current scrapers over and over again. At least the creator of PhantomJS is responsive. I will try LXC containers.
    – user3916597
    Commented Oct 19, 2016 at 20:55
  • a dead-man switch, maybe?
    – Robert Harvey
    Commented Oct 19, 2016 at 20:59
  • My dead man switch is configurable scala duration defaulting to 60 minutes which tends to be a general estimate for when things go sour. That would really require containers then. I will try the container approach, darn.
    – user3916597
    Commented Oct 19, 2016 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.