How to design a website with search capabilities?

Question

Suppose you want to create a web page with some search capabilities over a database. Search has to be restricted both on some categorization of data and open text given by the users. Functional and non-functional requirements pose this constraints:

• Search has to be fast
• Data representation paradigm (i.e. Relational, Object-Relational, Semistructured...) should support big modifications on every stage of software lifetime.
• Database's paradigm should permit easy refactoring for existing client code

Having this into account

1. Would you recommend a relational database? How would you tackle the constraints in this scenario?
2. Would you recommend a XML database? How would you tackle the constraints in this scenario?
3. If you think above solutions to be not useful in the light of this problem... what could be a useful approach?

Please give some bibliography (or some websites) supporting your answers.

thanks!

Answer 1

Lucene Index

• http://en.wikipedia.org/wiki/Lucene
• https://meta.stackexchange.com/questions/39950/how-is-lucene-net-utilized-on-stackoverflow-if-at-all
• http://blog.stackoverflow.com/2011/01/stack-overflow-search-now-81-less-crappy/

Generally speaking, some of the big website (think server and database clusters) applications that I've written, I've used a Lucene/Lucene.NET index to create a search index of appropriate database fields. The main advantage of this is speed, an in memory db index as opposed to a round trip -> db -> webserver for a search.

From a code development perspective lucene is completely separate from your database implementation and so you have a separation of concerns between how you want to represent your data, and how you index your data for easy searching. You can use a NoSQL db, Relational DB, etc. The search engine wont care as it's only getting its results from the index, and only when you need to display the actual contents you can grab the contents by an indexed field on the db and you're done.

See the search engine on:

• BigPondMusic

There are actually 2 lucene indexes behind the scenes, 1 index which is created specifically for predictive text (results returned are based on most popular searched on artist at the time), and the full db index which indexes albums, artists which match the search criteria.

So far the performance of the Lucene Index is exlemprarly, I have > 1 million db entries indexed by artist and album and can return N-Gram results data back < 1ms for the predictive text index.

Answer 2

Whether or not to use SQL or Lucene for searching depends on how sophisticated you need for your searching to be.

• If you don't need keyword based search, then stick with SQL.
• If you are interested in a more intelligent ordering of the results, then use Lucene.
• If you need keyword based search across a large amount of data, then use Lucene.

If you do decide to go with Lucene, then Embedding Lucene in Your Service gives advice to developers on how to start. You will still keep your data in SQL but you will also keep another copy of the searchable parts in Lucene. There is more here on SQL vs Lucene. It also addresses how to use Lucene in multi-threaded environments.