I'm strongly considering a web search engine for my next pet project. All the basic principles are clear, but some details are not. Namely, I can't find a neat way to search by exact match, e. g. "The Who".
This requirement imposes limitations on how I can modify the source text:
- I can't use a list of stop words (high frequency, low relevance words like prepositions - they might be very relevant in some corner cases!).
- I can't use stemming (converting words to their basic lexical form). If I do use it, I can't tell "program" from "programmed" and so on.
Not being able to cast all words to their basic forms and to throw away common low value words means a much, MUCH larger index. And worse, for the common search queries that do benefit from stemming, how do I even implement it? The only solution I see is keeping two indices - one with actual words and one with stemmed words, but having to keep two copies of the Internet is not really a solution - one is already plenty to make me scratch my head.