I'm trying to build a very small, niche search engine, using Nutch to crawl specific sites. Some of the sites are news/blog sites. If I crawl, say, techcrunch.com, and store and index their frontpage, then within hours my index for that page will be out of date.

Does a large search engine such as Google have an algorithm to re-crawl frequently updated pages very frequently, hourly even? Or does it just score frequently updated pages very low so they don't get returned?

Also, how can I handle this in my own index?

  • Interestingly enough, Matt Cutts (head of Google webspam team) talked about this kind of thing literally a couple of days ago. Only a brief overview, but it may be of some use to you - searchengineland.com/… – Anonymous Apr 26 '12 at 12:23
  • It's all statistics based. The engine continuously updates its database dedicating time proportional to its popularity, meaning unclicked sites are updated rarely while latest news sites are always up to date. If I were you, I would average the proposed expiration time with the click rate or something of that sort, but then it's up to you. – Neil Apr 26 '12 at 15:52
  • @Neil Understood. My engine is really simple and at the moment I don't have the time nor resources to crawl frequently enough to have the freshest content. For it's intended purpose, my engine also doesn't need the freshest content. However, it's important that it NOT point to pages with content that's changed since my indexing. I've come up with one easy (although very imperfect) solution: Count the number of /'s in the url and boost pages with higher / counts. It's a horrible solution, but for the major blogs and news sites I'm crawling, this might work with their deep site structure. – OdieO Apr 26 '12 at 21:45
  • Why haven't you given expiration time more thought? It's normally used to inform browsers when they can no longer use what is contained in cache but would have to again request from the server since it's likely changed since then. Of course, some sites have negative expiration times, meaning you must update always. You likely don't want to invalidate the page the moment you cache it, so you should find some sort of compromise, say a minimum expiration time of a day if proposed expiration time is less. – Neil Apr 27 '12 at 8:36

Yes, Google does manage the frequency of how much time it's crawlers are visiting a site. For example all the news site which are registered on Google News get more frequent visits than other sites, also some live-score showing websites get crawler's vistit every second(realtime search). So, it means they are using some kind of algorithm so that those bots prioritise their visits to such sites.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.