How important is index size when searching?

Question

My company has recently began using Apache Solr to search its data. As we learn how to use it we have gone down the path of indexing multiple fields to get the results we need. Most of these are either N-Grammed or Edge-N-Grammed (N-grammed, but only from the edge; for starts-with searching).

Gramming by nature takes up a lot of space, which takes more time to search. Space is cheap, but time is less so. Index time is not too important, since a delta-import (only get the changes since last index) is extremely quick and you only pay a penalty on the first import. What we've not been able to determine is what effect the index size has on query times. Obviously a larger index takes longer to search, but the time added by n-gramming a field is difficult to predict.

How do you determine whether a field is worth gramming? Can you predict how much longer a query will take when you gram a field?

Answer 1

Well, the best thing you can do is run your own benchmarks!

Compare the average speed over several thousand test queries on an index that includes ngrams vs one that excludes them. It doesn't have to be your full actual index (since that may take a long time to generate), just a large enough sample size to get an idea.

Note that you can use debugQuery=on to analyze how a query is performed and maybe to generate a better index. For example, see Hathi Trust's Digitla Library Tuning Search Performance. By using CommonGrams and analyzing queries they were able to reduce average query time by 50%.

So, running your own benchmarks is best, but there are also some existing benchmarks online. For example see Sakai Solr Benchmark, which compares—among other things—the query performance using an index with ngrams and one without. If the details are similar enough to your use case, this benchmark should give you a rough idea of how it will turn out for you. To summarize this benchmark:

• Using documents about 2000 words in length and composed of random English words at English-like frequency. Total corpus about 6gb.
• Limiting the indexed n-grams to 3-, 4-, and 5-grams (also with one test indexing left-anchored edge n-grams up to length 15).
• Performing benchmarks under load of 5 concurrent users.
• Using average to sub-par server hardware.

Results (in average query time):

• Without n-grams: 159ms
• With 3-, 4-, and 5-grams: 393ms
• With 3-, 4-, and 5-grams and left-anchored edge up-to-15-grams: 450ms

(They also have some other results including what they are calling "lean" indexes).

The takeaway: If your data is similar enough to theirs, adding n-grams to the index may increase your query time by a factor of 2.5 to 3. Of course, you have to take these results with a grain of salt, because there are so many factors specific to your data. This is best used not as a fact, but as a guideline for what to perhaps expect when you run your own benchmarks.