0

For one of the projects at uni we were given the task to create a custom, niche search engine. My colleges and I split the tasks amongst each other so that we can tackle the overall project more easily. My part is to create the indexer. I already read the wikipedia page on search engine indexers and some other related articles but I'm still struggling to understand exactly how it works and how it looks.

To me it is obvious that it is not just a regular table with an index and a descrption column. So my question would be, what is a search engine indexer comprised of, how does it architecture look, and where to start from in building one?

|improve this question
  • 5
    To me it is obvious that it is not just a regular table with an index and a descrption column Actually in it's simplest form that's about it. – yannis Mar 20 '13 at 9:05
  • 2
    This would depend on your project requirements (and have you to do everything yourself or can you use tools?). A database would, as Yannis says, be the most simple form. Some full text search engine (Apache Lucene) would be smarter and worth to look at if you need to do it yourself. Beyond that you could evaluate where in the html a piece of text was found (header is worth more than paragraph etc) This all is of course still a few steps away from full Google smartness... – thorsten müller Mar 20 '13 at 9:08
  • We are not only allowed, but also encouraged to use third party tools in creating the niche search engine. My question is what does the architecture of an search engine indexer look like? – Dragan Mar 20 '13 at 9:13
  • 1
    @Dragan Please update your question to tell us what kind of data you will be indexing, what infrastructure you have available (language & database, for example) and every other relevant information (like your previous comment). The specific technologies aren't that important, but the architecture might differ significantly based on the kind of data you are targeting. – yannis Mar 20 '13 at 9:16
  • 4
    @Dragan: a "search engine indexer" can be whatever you want to it be, so get the requirements clear. Start with defining an API for your indexer - what exactly do your colleges want to give your component as input, and what is expected to be returned? For example, is it expected to input just a single word, and your indexer shall return a list of URLs + positions for that word? Or do you have to evaluate a complex search string? How are the crawler results are fed into your indexer? Describe that first before thinking about "how it works". – Doc Brown Mar 20 '13 at 9:56
7

At its core, a search engine index is simply an index that supports full text search. The most simple way to do that is a simple inverted index, i.e. for each word that occurs in any of the documents you have indexed, store a list of references to all the documents that contain this word.

For a university project, that's probably enough, but of course there's infinite room for improvement. You can combine multiple search words using AND and OR logic, and have a weight for each document depending on where and how often a word appears in it. That's the state of WWW search engines circa 1998, before Google revolutionized it with their PageRank algorithm. Since then, they've had hundreds (if not thousands) of people working on improving it continuously.

Additionally, to support an index for the entire WWW (or even a small part of it), you need a distributed architecture, something like MapReduce.

|improve this answer
  • 2
    Google was a university project! – Sam Mar 20 '13 at 9:18
  • 2
    @Sam: ...of Ph.D. students. – Michael Borgwardt Mar 20 '13 at 9:22
  • 1
    Just to be on the safe side clear, the information provided here is only in connection to the indexer part. The web crawler part will be done by a fellow student – Dragan Mar 20 '13 at 9:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.