1

Researching archiving systems like archive.org, found out main issue of such is the dynamic content.

Initial analysis shows that content 'dynamicity' can be assigned to one of the following levels:

  1. Static html content - plain old web page which is represented only by html markup with auxiliary css-referred resources (usually images).

  2. Static html powered by javascirpt – same as Level 1, but has javascript code, which only manipulates existing markup (such as expand/collapse).

  3. “Onload” page construction – web page with javascript code, which makes a certain additional requests during page load phase. After loads phase page content is fully constructed.

  4. Dynamic client-side content – UI elements are modified by javascript code on-the-go, as user traverses through interface. Usually these are modern SPA (single-page-applications, like gmail.com), “endless” lists (list tail is loaded when user scrolls down to the list bottom) , loading content on demand (smart expanders) and so on.

So I assume that Levels 1 and 2 can be archived pretty easily. Could you please suggest how to handle Levels 3 and 4? Looks like it should involve page rendering, but some details would be helpful.

Update: To clarify the question: ideally offline version should be fully-functional, at least within the site level (ignoring external domains content). Also, if Level4 is too hard to automate fully - is there an approach involving human operator who makes hints to the system about content?

Improve this question
3
  • Handle level 3 and 4 to do what exactly? If you want to store them it would depend what you want to do with that data. Activating the script in what you call level 3 would be tricky but maybe possible, in level 4 I don't think there is a really good way to 'simulate' user interaction (unless you write this per site)
    – thorsten müller
    Commented Jun 24, 2013 at 7:42
  • @thorstenmüller: I've update question. Ideally offline version should be fully-functional. If there is not way to avoid user-aided process, so be it: there could be semi-automatic approach.
    – mikalai
    Commented Jun 24, 2013 at 7:48
  • I don't think that there is a way to make content easily available that the page requests with javascript, not without a lot of programming work anyway. But even then you would need to put the content into a local server that somehow simulates the response to requests from that site plus makes sure the sites requests are redirected to that server. I don't know if you are aware how exactly javascript is used to generate and process such requests, but a site can send a large amount of information as params and the response can be different for each combination.
    – thorsten müller
    Commented Jun 24, 2013 at 8:18

1 Answer 1

Reset to default
0

It is doable, integrate in your crawler a webkit browser. Then get all static pages first. Log down the requests made by the page (you can now because the browser in your crawler actually renders the page).

That will give you an overview of the responses done.

The onloads are easy because they are done directly. The harder part is clickable elements which load additional content. To get them find everything with a custom event attached and execute the event to see what happens. If content changes you will know.

The responses could be cache so you could create a full working version.

Considerations are mostly processing time and browser issues. It is much slower to crawl this way then textual parsing since you actually need to load and render the page.

Improve this answer
3
  • You still would have to host the results on the server in a way to simulate the response? Otherwise you would have to 'replace' parts of the javascript? Anyway, there may be things that you can click that influence the javascript request (like checkboxes to select params for a search request). But yes, something like a webkit browser as part of the crawler would be a way and there should be some tools to help with that.
    – thorsten müller
    Commented Jun 24, 2013 at 10:41
  • Thanks. The question is about js calls - if they go to 'site.com/getdata', is there a way to intercept the call in an archived page and set the url to an archived data? How do you determine if an element is 'clickable'?
    – mikalai
    Commented Jun 24, 2013 at 22:37
  • If you use a webkit browser with the developer tools activated you see in the tab: Net that the requests are available. So if you can communicate in any programmatically way you can intercept them. Because it is open-source you could even attach some custom code on that. There might also be JS examples (which you could include on top of page) like: stackoverflow.com/questions/8251955/… which you could research. In general: Parsing the page by yourself will be impossible so you need some kind of browser library.
    – Luc Franken
    Commented Jun 25, 2013 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.