I have built a very basic webcrawler running off my laptop so it has limited memory and limited hard drive space. The way I have it now is I'm using MongoDB to store the links I find on pages. I make sure to sanitize them and remove any unnecessary URL parameters that might cause duplicate pages to be fetched. Every time I visit a webpage, I simply grab all the links, sanitize them, and then do the following....
For each link, I check in the database if it has already been created (created, not necessarily visited). If so, then I move onto the next link. If not, I create the link. So this happens one by one for each link. I also flag it as unvisited.
After that step I grab one unvisited link, fetch its links, and mark it as visited. That's pretty much it. It then repeats this whole process for each link.
The problem is that large subsets of pages have hundreds of the same links, as from templates perhaps, like categories or tags or search result pages. So what I am seeing in the logs when I grab the links off of a page I visit are a bunch (100+ quite often) of "link found already in database", per fetched page. This adds several seconds on my laptop to each page visit. Most of my logs are showing that "link found already in database" type of thing. After several hours of running it, I've only managed to collect a few thousand unique links in some cases. I would've expected to be in the millions of links by now.
I am wondering how to make this faster, improve the overall architecture to get better performance and coverage of the crawler over the pages, so it doesn't keep checking the database 100's of times per page.
The first thing I thought of was to use an in-memory Trie of the URLs, and so (instead of checking the db) you could quickly check in memory if we have the URL. But that approach ran into an "out of memory" error pretty quickly, since I am running several separate crawlers at once on my laptop and have other apps running and using up the memory.
So I'm left wondering how you can go about fetching the links off a page, only saving the links that are new, and yet not wasting a lot of time checking if the links already exist. I am wondering how large-scale production systems might architect a solution to this. I could imagine throwing hundreds of computers at the problem, but that is cost-prohibitive. Wondering if there is anything else. I would like to learn how it's done.