I started building a web crawler and read somewhere that it's a very hard problem to find a good seed page for the crawler. Can anyone explain me if there is any pre-defined procedure/ guidlines of finding a good seed page? or how you say that a particular page is a good seed page?
closed as off topic by Blrfl, user7007, Martijn Pieters, user40980, GlenH7♦ Feb 1 '13 at 15:07
Questions on Software Engineering Stack Exchange are expected to relate to software engineering within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here. If this question can be reworded to fit the rules in the help center, please edit the question.
A good seed page needs to have
- As many links as possible
- For as many different topics as possible
That's all, really. The first things that come to mind would be Wikipedia and the Open Directory Project.
Since I believe generalistic web search should be now considered a solved problem with a single provider that has such an headstart, and momentum, that it's unpractical, unconvenient, and kinda lame to even consider outrunning it, I believe you might prefer focusing your search efforts on a specialized domain.
e.g.: U.S. law, E.U. law, local laws of certain foreign countries, documents in a certain language, fishing, hunting, the military, ambientalism, indian car manifacturers, south american team games, russian motorsports...
Specializing your domains is all about finding the appropriate seeds for the given domain. (Also, IMHO, a handul of selected, unknowing, specialized forums should also be considered as a starting point) So once you have you domain, if you dig it well, (or at least the one you're working with/for does) the seeds should come natural.
Otherwise head for dmoz.org like Google did some 15 years ago. (And don't forget to weight in yahoo.com for good measure) Give your crawler another 15 years and you should have a good link base to search upon.