Getting data from a webpage in a stable and efficient way

Question

Recently I've learned that using a regex to parse the HTML of a website to get the data you need isn't the best course of action.

So my question is simple: What then, is the best / most efficient and a generally stable way to get this data?

I should note that:

• There are no API's
• There is no other source where I can get the data from (no databases, feeds and such)
• There is no access to the source files. (Data from public websites)
• Let's say the data is normal text, displayed in a table in a html page

I'm currently using python for my project but a language independent solution/tips would be nice.

As a side question: How would you go about it when the webpage is constructed by Ajax calls?

EDIT:

In the case of HTML parsing, I know that there is no actual stable way to get the data. As soon as the page changes, your parser is done for. What I mean with stable in this case is: an efficient way to parse the page, that always hands me the same results (for the same set of data obviously) provided that the page does not change.

Answer 1

Well, here are my 2 cents:

If there is no AJAX involved, or it can be cleared easily, 'fix' the HTML to XHTML (using HTMLTidy for example), then use XPath instead of regular expressions to extract the information.
In a well-structured web page, the logically separated entities of information are in different <div>s, or whatever other tag, which means you would be able to easily find the right information with a simple XPath expression. This is great also because you can test it in, say, Chrome's console, or Firefox' developer console and verify it works before writing even one line of other code.
This approach also has very high signal-to-noise ratio, since usually expressions to select the relevant information will be one-liners. They are also way easier to read than regular expressions and are designed for that purpose.

If there is AJAX and serious JavaScript-ing involved in the page, embed a browser component in the application and use its DOM to trigger events you need, and XPath to extract information. There are plenty good embeddable browser components out there, most of which use real-world browsers under the hood, which is a good thing, since a web-page might be incorrect (X)HTML, but still render good on all major browsers (actually, most of the pages eventually get this way).

Answer 2

In my experience, using .NET environment, you can take advantage of HTML Agility Pack.

If the page is formatted as XHTML you can also use a regular XML parser. There's a lot out of there for any environment you can imagine.

For the side question about AJAX, you can use regular HTTP networking code to get data and parse it.

Again if your AJAX stack returns XML, you'll got a lot of choices. If it returns JSON, consider a library that allow you to map the stream to typed objects. In .NET I suggest you Newtonsoft.Json.

Answer 3

Parsing HTML is not a completely trivial task, since one has to deal with possibly incorrect markup (tag soup). During the years, browsers have implemented more or less the same strategy to deal with errors, and that algorithm has been christened in the HTML5 specification (yes, the HTML5 specification specifies what to do with things that are not HTML5).

The are libraries for all major languages to parse HTML, for instance this one.

In any case, what you will get is not stable in any sense. Each time the webpage format changes, you have to adapt your scraper.

Answer 4

As a side question: How would you go about it when the webpage is constructed by Ajax calls?

If ajax calls are being made, then its very likely its either some POST or GET url with some variables.

I would examine the JavaScript to find out what the endpoints and parameters are. After that its very likely that either the data returned is json/xml/plain text or perhaps partial html.

Once you know the above information, you simply make a GET or POST request to that endpoint, and parse the returned data.

Answer 5

There is no stable or better way to do this, HTML web pages was not made to be manipulated by computers. It is for human users, but if you need to do it I suggest that will use a browser and some javascript. At my work I was involved with a project that need to extract some information from a third party site. The application was developed as a Chrome extension. The application logic is written using javascript that is injected on the site after the Page load is complete. The data that is extracted is sent to a database through a http server. It is not the best approach, but it works. Ps: The Site owner has authorized us to do such thing.