What is a good approach to deliver highlighted search results in a full text database?

Question

Let's say we have a full-text database. The reasonable thing to do, for scalability, is to store only an index but not the content of documents.

The issue I'm facing is: I would like to deliver highlighted preview of the content when listing search results for a given query (just like Google does), but I find hard to choose one of the approaches over the other:

Approach I:

1. Creating an index for the documents set.
2. Storing the text content of the documents.
3. After retrieving documents for a given search query, a highlighted preview is created using the previously stored content of documents.

Issue: storage.

Approach II:

1. Creating an index for the documents set, without storing their text content.
2. After retrieving documents for a given search query, a highlighted preview is created by dynamically (at query-time) reading the text content of the documents.

Issue: it may take a considerable amount of time to parse the text content of documents while the user is waiting.

So, are there any known approaches to this problem? Otherwise, which one do you think, relatively, costs less than the other?

Answer 1

I'd suggest you use ElasticSearch highlight feature or taking a look at how it works internally. With settings, it supports breaking a text into fragments and find the best fragments among all fragments. Finally it highlight only those terms that participated in generating the hit on the document.

Answer 2

You can store the index and the first couple of highlights and its surrounding text per item(as their preview).

By this way looking at the initial results user can decide to retrieve the document or not.

This approach will definitely increase your storage but not as much as the your first approach. It will be though more or less at the same time if your preview options are limited to a paragraph or so.

Answer 3

Depends on your distribution of indices:

Thake the following example sentence:

The government passed a bill on tax deductions

If every single word would be assigned an index and a context you would be storing the context quite a number of times (assuming there are 4 keywords in this sentence (the nouns), you would probably end up storing the whole sentence with each of the four keywords.). Defining yet another indexed keyword from the sentence would store the whole sentence even one more time.

Keep the context in the file, store the offset of occurrences in the file with the index, load and display surrounding text directly from the text file when presenting search results. (That's basically your approach II).