I would like to create a site where users can post articles with the following optional parts:

  • A title
  • Contents (text)
  • Categories
  • Keywords

Articles will be stored in mongodb and the site will be built in node.js. Users can search the site using a normal search text box.

I'm thinking about creating the following collections:

  • Users
  • Articles
  • Keywords

I will then create an entry for each keyword used in the Keywords collection with an array containing all the articles that use it. If a user conducts a search, the search is broken up into keywords and each keyword is looked up in the Keywords collection. Each article is then retrieved from the db and ranked based on relevance.

My questions are:

  1. Would it be efficient to use a Keywords collection like this, should I just use the Articles collection (Using full-text search or something) or should I structure it in some other way?
  2. How would I incorporate the ability to search the title, contents or categories for articles instead of just the keywords?
  3. Would it be better to use something like Apache Lucene than to build this functionality myself?
  • 2
    3.) Most definitely – Falcon Sep 23 '13 at 14:19
  • I would use Apache Solr / Lucene out of pure laziness, plus it will provide you with lots of features difficult to implement on your own. – thorsten müller Sep 23 '13 at 14:30

I would use an existing platform designed for search. You mentioned Lucene and there are others around based on the language you are using.

If you want to create a stand alone search server that is language agnostic look at SOLR. It is based on Lucene, so lots of support.

I personally like Sphinx, but it may not work in your situation, it all depends on the language you are using.

| improve this answer | |

It would not be a good idea to build the search as you suggest for a number of different reasons.

  1. You are relying on the creation of useful, accurate keywords, and the management of those keywords. How are new keywords created? Who is responsible for linking the right keywords to the content?
  2. You don't have clear mechanism for ordering the search results. How would you decide which of the articles might be the most relevant? If only a keyword is being used, all articles would have the same weight so you would have to default to something like recency or alphabetical order by title.
  3. You are ignoring a lot of useful information about the content in the search. The title and the body of the document will contain some information that is not contained in the keywords, unless someone spends a lot of time managing keywords.

Keywords can still play a role, but perhaps as one of many fields (including title and content) used in the search. They could be simply be managed as an array in the main record for the article rather than a separate collection that needs to be maintained and linked to the content.

Given that you are committed to using MongoDB, I would suggest trying MongoDB's native full text search capabilities as a start. Try indexing the title, content, and keywords (stored as an array with the article). You can play with the weights applied to each column to determine how they impact relevance.

As @Bill Leeper suggests, standalone search engines could also be an option if you find you need something more sophisticated than what is built into MongoDB, or MongoDB's relatively new full text search doesn't work as expected. SOLR could be a good starting point because it is language independent and very rich.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.