Do you know any good tutorials, frameworks, anything that can help me to write code that captures information from a website that don't have a public API, or hasn't been written in a RESTful way?

I've heard there's a Ruby gem that simulates a browser, but I forgot its name and couldn't find any information about it anymore.

  • 1
    This is contradictory--the title asks about non-RESTful sites, then you ask about websites "written in a RESTful way". Which is it? – Matthew Flynn Mar 21 '12 at 20:54
  • Sorry! It should be 'hasn't been written in a RESTful way'! ;) – João Daniel Mar 21 '12 at 21:02
  • You can look into HTTrack to see how they do it. – sylvanaar Mar 25 '12 at 0:14

There are a variety of Ruby libraries for parsing (X)HTML into a structured format. Two of the oldest and most popular libraries for this are Hpricot and Nokogiri. These libraries provide a variety of capabilities for working with HTML data.

A few things to keep in mind when doing this:

  • Respect copyright! site authors likely won't be happy about people scraping data from their sites and using it without permission. Ask first!
  • Changes can break things easily. If they change the structure of your site, your code will likely blow up if not written properly.
  • 2
    +1 for those two bullet points, which you absolutely must keep in mind if doing anything like this. – Carson63000 Mar 21 '12 at 22:40
  • I plan on index prices from online stores. Is it something copyrightable? – João Daniel Mar 21 '12 at 23:30
  • @JoãoDaniel: Technically they can stop you, either by legal or technical means (eg. constantly making subtle changes). But they probably won't IF you are offering some value to them by doing whatever you're doing. Look to the biggest player in the market; they are the one who won't appreciate you advertising other companies to their customers. If you can convince them it's in their interest, you'll be fine; otherwise don't waste your time. – pdr Mar 22 '12 at 1:04

Well, for Java you can use the apache HttpClient library, which will allow you to (easily) make http calls and cope with sessions and the like. Likewise, .NET has an class called... HttpClient.

Consuming the response, if you don't have some sort of predefined arrangement, will require you to set some expectations (it can't really be TOTALLY arbitrary) and do some parsing to glean information from the response you get.

I can't speak to Ruby, but I'm guessing there's something similar in most modern languages.

Not the answer you're looking for? Browse other questions tagged or ask your own question.