I want to create an HTML scraper in Ruby on Rails and I want to implement a progress bar and an ability for it to pick up where it left off if it's interrupted while scraping.

I think the best way to do this be to create a database table with a record for every single page, with a boolean column called 'scraped?'

I'm thinking my scraper would have two main parts. The first part would loop through the entire site, following all links and saving their URLs in the table, along with a false for their scraped? attribute. It would also reset the database's pk sequence, meaning the id column would accurately imply which number the record is at (more about that later!)


The second part would then move through this table, visiting each link and, once it's done scraping that page, it would change its scraped? attribute to true.

This second part would start by getting the first page that hasn't been scraped yet:

first_page = Page.where(:scraped? = false).first

The progress bar would work on the following principle:

total_page = Page.count
done_pages = first_page.id # (the first result would always be 1 because the first part resets the sequence)
percentage = done_pages / total_pages * 100

I actually don't see why this wouldn't work, but I'm still really vague about how the scraper would spider over and visit every single link on a website.

What would be your ideas on this? I just want some pointers before I dive in:

How would it know not to stray off the current site? (I'm thinking regex on the URL to make sure it contains the specified domain before it visits it)

How would it not visit infinite loops? Such as a link to the home page on the home page? (I'm guessing a conditional would do the trick here):

if new_uri != current_uri
    visit new_uri

How would the first part actually go about its spidering, saving an index of every single link? By creating an array of every single link on the page and then iterating through it until it finds a uri that doesn't exist in the database? (this is actually my biggest concern.)

If the above is true, how would it know when to stop? I'm guessing it would check every new_link with existing links in the database? This is only a local program that's going to be executed once so I'm not too concerned about performance here, but do you have a better idea to the below:

if new_uri != current_uri # technically not necessary but it would save a few database interactions
    if Page.exists?( uri: new_uri) == false
        visit new_uri

It's actually much harder than you've described:

  1. Some (different) links may refer to the same page. Consider /forum/index.html?topic=3&page=2 and /forum/index.html?page=2&topic=3 as a trivial example (request parameters are swapped). So detecting if page is crawled already would require you to check page contents.
  2. Page contents may change on every request. Consider "posted 1 minute ago" on comment post.
  3. Some pages may be inaccessible for many reasons (so URLs must be marked accordingly):
    • link is broken
    • authorization required
    • page no longer exists
  4. Some URLs may be "internal", but actually redirect to other sites: http://vk.com/away.php?to=http%3A%2F%2Fexample.com%2F
  5. Some URLs may lead to files, not HTML pages

Summing it up, you must be able to:

  • set "states" on URLs: "checked"/"not checked", "malformed", "access denied", "outside redirect", etc. - you may actually use HTTP response statuses here
  • store MIME type of response to distinguish HTML pages from downloadable files
  • make diffs on pages' content, so you can tell if two URLs point to the same page. Simple textual diff probably won't work because HTML structure must be respected. Definition of "same page" may be complicated and depend on how much of content is different (e.g. pages are considered to be the same if >95% of plaintext is the same and HTML structure is the same)

It's probably even more complicated than I think, because I never made a crawler myself. First thing to do here is find some existing crawlers and either using one of them or examine them as examples of how things whould work. Check out gems (if it has to be Ruby) like Mechanize to start (just found it after 2 minutes googling, so google deeper).


Not the answer you're looking for? Browse other questions tagged or ask your own question.