0

Let's say we have a full-text database. The reasonable thing to do, for scalability, is to store only an index but not the content of documents.

The issue I'm facing is: I would like to deliver highlighted preview of the content when listing search results for a given query (just like Google does), but I find hard to choose one of the approaches over the other:

Approach I:

  1. Creating an index for the documents set.
  2. Storing the text content of the documents.
  3. After retrieving documents for a given search query, a highlighted preview is created using the previously stored content of documents.

Issue: storage.

Approach II:

  1. Creating an index for the documents set, without storing their text content.
  2. After retrieving documents for a given search query, a highlighted preview is created by dynamically (at query-time) reading the text content of the documents.

Issue: it may take a considerable amount of time to parse the text content of documents while the user is waiting.

So, are there any known approaches to this problem? Otherwise, which one do you think, relatively, costs less than the other?

3
  • 5
    Text search engine (e.g. elasticsearch) can highlight search terms out of the box. If you find your database index is lacking features to meet your requirements, I would recommend using a dedicated search engine product.
    – Samuel
    Sep 29, 2017 at 21:48
  • Not getting the big deal of storing the text of a document in the DB.
    – paparazzo
    Nov 28, 2017 at 22:35
  • 1
    Highlight it on the front-end (forget about the highlighting for the purposes of search). Jun 27, 2018 at 13:13

3 Answers 3

2

I'd suggest you use ElasticSearch highlight feature or taking a look at how it works internally. With settings, it supports breaking a text into fragments and find the best fragments among all fragments. Finally it highlight only those terms that participated in generating the hit on the document.

1

You can store the index and the first couple of highlights and its surrounding text per item(as their preview).

By this way looking at the initial results user can decide to retrieve the document or not.

This approach will definitely increase your storage but not as much as the your first approach. It will be though more or less at the same time if your preview options are limited to a paragraph or so.

2
  • 1
    As a free text search might use any word or combination of words, this would mean storing full text. Only storing context on some keywords might result in search displaying no keyword in context at all.
    – Grimaldi
    Apr 3, 2017 at 7:19
  • @Grimaldi since reaffer started with an index, I assumed that is his way he would like to go, or a special case. Apr 4, 2017 at 14:27
1

Depends on your distribution of indices:

Thake the following example sentence:

The government passed a bill on tax deductions

If every single word would be assigned an index and a context you would be storing the context quite a number of times (assuming there are 4 keywords in this sentence (the nouns), you would probably end up storing the whole sentence with each of the four keywords.). Defining yet another indexed keyword from the sentence would store the whole sentence even one more time.

Keep the context in the file, store the offset of occurrences in the file with the index, load and display surrounding text directly from the text file when presenting search results. (That's basically your approach II).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.