I want to create an HTML scraper in Ruby on Rails and I want to implement a progress bar and an ability for it to pick up where it left off if it's interrupted while scraping.
I think the best way to do this be to create a database table with a record for every single page, with a boolean column called 'scraped?'
I'm thinking my scraper would have two main parts. The first part would loop through the entire site, following all links and saving their URLs in the table, along with a
false for their
scraped? attribute. It would also reset the database's pk sequence, meaning the id column would accurately imply which number the record is at (more about that later!)
The second part would then move through this table, visiting each link and, once it's done scraping that page, it would change its
scraped? attribute to true.
This second part would start by getting the first page that hasn't been scraped yet:
first_page = Page.where(:scraped? = false).first
The progress bar would work on the following principle:
total_page = Page.count done_pages = first_page.id # (the first result would always be 1 because the first part resets the sequence) percentage = done_pages / total_pages * 100
I actually don't see why this wouldn't work, but I'm still really vague about how the scraper would spider over and visit every single link on a website.
What would be your ideas on this? I just want some pointers before I dive in:
How would it know not to stray off the current site? (I'm thinking regex on the URL to make sure it contains the specified domain before it visits it)
How would it not visit infinite loops? Such as a link to the home page on the home page? (I'm guessing a conditional would do the trick here):
if new_uri != current_uri visit new_uri end
How would the first part actually go about its spidering, saving an index of every single link? By creating an array of every single link on the page and then iterating through it until it finds a uri that doesn't exist in the database? (this is actually my biggest concern.)
If the above is true, how would it know when to stop? I'm guessing it would check every new_link with existing links in the database? This is only a local program that's going to be executed once so I'm not too concerned about performance here, but do you have a better idea to the below:
if new_uri != current_uri # technically not necessary but it would save a few database interactions if Page.exists?( uri: new_uri) == false visit new_uri end end