What sort of languages would be able to handle writing your own web crawler?

Could PHP handle this? I'm quite good with PHP (following best practices etc).

But I'd like a good reason to learn a new language if I need to.

The idea is to crawl one specific website that has multiple entries, much like an RSS feed, but they don't offer that an RSS feed of the site...

  • 6
    Don't forget to download the robots.txt file and obey the site restrictions and crawling speed limits. Websites can get quite annoyed when you go faster and start banning you. en.wikipedia.org/wiki/… Aug 2, 2011 at 2:04

2 Answers 2


Any language that has good built-in HTTP facilities for downloading and parsing (badly formed) web documents would be fine. I'd recommend a scripting language like Python, because a webcrawler generally spends most of its time waiting on network I/O (DNS or HTTP responses), so performance is generally not much of a concern. Python also has decent libraries for managing HTTP connections and parsing HTML. You need a library specifically designed to handle badly formed HTML, because the web is mostly crappy HTML. If you choose Python, you can use something like libxml2dom.

  • 4
    Wait, what, performance is not a concern? Sure, you'll be doing a lot of I/O waits, but you'll also probably be processing 50 requests in parallel.
    – Aaronaught
    Aug 2, 2011 at 2:13
  • Even with 100 parallel requests using a combination of async I/O and multiple threads, a typical webcrawler spends the majority of its time waiting for a response from remote webservers. Aug 2, 2011 at 5:16
  • 1
    Well, you're correct of course that it depends on what you're doing with each fetched document. But unless you're doing something incredibly CPU-intensive with each download, you're not likely to be CPU bound for any significant percentage of the crawling time. In practice the CPU will be sitting idle a lot, even with hundreds of parallel requests, because the crawler not only has to wait for responses from each remote server, but also implement artificial wait times for politeness purposes, e.g. wait 1-5 seconds per request per host. Aug 2, 2011 at 13:39
  • 1
    So, unless you're pounding a single host over and over, you're not likely to be CPU bound for any significant portion of the webcrawl. Even a commercial crawler that fetches from thousands of hosts simultaneously will wait a long time, since most hosts have few documents and the crawler will eventually end up slowly fetching docs from a few large hosts. That's been my experience, at least. Aug 2, 2011 at 13:40
  • 1
    @Aaronaught: I write a crawler for my company. IO is definitely the primary bounding factor (even without DNS lookup done by a separate scheduling processes). Parsing and correcting for HTML is trivial in comparison. We spin up-to max 20,000 parallel requests in each crawler and have 10 crawlers running simultaneously. And yes it can go very paralleled, but you must obey the speed-limits on each site (and even if they do not have explicit speed-limits we make sure we do not hit sites hard (as we don't want to banned)). Note: multiplex over 20,000 sites is hard so this limit is rarely reached. Aug 2, 2011 at 14:33

I would recommend you using python, but that's because I only know how to -code- a crawler in python and Java. And I would definitely not recommend you going 5 feet near Java for that task.

Take a look at:
Mechanize - good tool for web browsing
Scrapy - good framework to screen scraping and web crawling

A good reason is that you already have some frameworks built in python just for that.

  • I agree, Python is going to be the fastest route to something working that doesn't explode on strange input.
    – user131
    Aug 2, 2011 at 7:28
  • Recommendation based on the fact that is the only language you know is not a good recommendation. Recommendations require that you have done it in many languages and know the pros/cons for each. Aug 2, 2011 at 14:40
  • that's why I added the comment... And I've done it in ruby, java, python and c#. But I've REALLY done it in Java and Python. If you go around saying that you're the master of the universe and know how to code it properly in all n languages.. Yeah.. People is going to believe in it...
    – wleao
    Aug 2, 2011 at 14:46
  • Scrapy looks nice. Just saying
    – MattyD
    Aug 2, 2011 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.