I have built a very basic webcrawler running off my laptop so it has limited memory and limited hard drive space. The way I have it now is I'm using MongoDB to store the links I find on pages. I make sure to sanitize them and remove any unnecessary URL parameters that might cause duplicate pages to be fetched. Every time I visit a webpage, I simply grab all the links, sanitize them, and then do the following....

For each link, I check in the database if it has already been created (created, not necessarily visited). If so, then I move onto the next link. If not, I create the link. So this happens one by one for each link. I also flag it as unvisited.

After that step I grab one unvisited link, fetch its links, and mark it as visited. That's pretty much it. It then repeats this whole process for each link.

The problem is that large subsets of pages have hundreds of the same links, as from templates perhaps, like categories or tags or search result pages. So what I am seeing in the logs when I grab the links off of a page I visit are a bunch (100+ quite often) of "link found already in database", per fetched page. This adds several seconds on my laptop to each page visit. Most of my logs are showing that "link found already in database" type of thing. After several hours of running it, I've only managed to collect a few thousand unique links in some cases. I would've expected to be in the millions of links by now.

I am wondering how to make this faster, improve the overall architecture to get better performance and coverage of the crawler over the pages, so it doesn't keep checking the database 100's of times per page.

The first thing I thought of was to use an in-memory Trie of the URLs, and so (instead of checking the db) you could quickly check in memory if we have the URL. But that approach ran into an "out of memory" error pretty quickly, since I am running several separate crawlers at once on my laptop and have other apps running and using up the memory.

So I'm left wondering how you can go about fetching the links off a page, only saving the links that are new, and yet not wasting a lot of time checking if the links already exist. I am wondering how large-scale production systems might architect a solution to this. I could imagine throwing hundreds of computers at the problem, but that is cost-prohibitive. Wondering if there is anything else. I would like to learn how it's done.

  • Why don't you use a hashmap to check for duplicate links?
    – Pieter B
    May 8, 2019 at 9:40
  • It will run out of memory, as there could be a billion links. May 8, 2019 at 9:43
  • You need to run a profiler to see why your webcrawler is so slow.
    – Pieter B
    May 8, 2019 at 9:53
  • The suggestion of hashmap of the addresses and compiling the database queries into a single call are good ones. If you're finding only a few thousand links and your starting source is something big like stackexchange, then there is probably some other error. You are ensuring that the link is new before going through its own links, right? And you're not pruning too much from the address? Jun 7, 2019 at 13:48

1 Answer 1


If you are hitting the DB multiple times per page, you are doing it wrong. Try sending the list of links to the DB in one query.

This will be a MERGE ... WHEN NOT MATCHED operation.

At some point in scaling, it is more costly to check that you haven't visited a link elsewhere than it is to just duplicate the visit. You don't need a perfect uniqueness check, if you have a locally good-enough one.

  • Even with using bulkWrite of updateOne from MongoDB it is taking upwards of 30 seconds to write 2300 links to the db. May 8, 2019 at 9:44
  • 2
    Have you got any indexes on your data? That sounds inordinately slow
    – Caleth
    May 8, 2019 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.