5

I'm talking the kind of search that auto-suggests your query as you type, the way Google does, the way Wikipedia does, the way Stack Exchange suggests other questions as you type the title, etc. And Wikipedia used to have an awful search implementation, for those of us who remember. It had no auto-suggest, was shut down by the slightest typo or misspelling, and rarely gave pertinent results.

As an amateur web programmer, I'm trying to figure out how I would implement an effective and high-performing search-and-auto-suggest as Wikipedia now has, and as SE has, and as so many other websites have.

How can you obtain that kind of incredibly fast performance and flexibility? Does everyone just wing it, or are there principles, guidelines, helpful libraries, or other such resources to help you get it done right in the style of the "Internet of the Future"?

Improve this question
8
  • Should this be a StackOverflow question? Sometimes I can't tell.
    – temporary_user_name
    Commented Jan 24, 2013 at 8:22
  • 2
    no, I don't think so. It's not specific enough for SO (roughly: can't be answered with code)
    – thorsten müller
    Commented Jan 24, 2013 at 8:25
  • Chances are the answers to this question do equally well apply if you produce a search for an application that stores a lot of data. It's probably not restricted to websites.
    – Joachim Sauer
    Commented Jan 24, 2013 at 9:41
  • Seems like a question for ux.stackexchange.com
    – JohnL
    Commented Jan 24, 2013 at 10:22
  • 1
    Ironically, I've never been crazy about the look and feel of that site... Feels 'over-designed' or something.
    – GHP
    Commented Jan 24, 2013 at 14:21

1 Answer 1

Reset to default
6

For a start you will need a full text search engine like Apache Solr or Sphinx (there are more and some databases have full text features too, but I know those two and they are free and work great). If it has facet search (like Solr) this will help a lot (for certain types of queries). This will cover the largest part of indexing and performance issues.

From there on you need to analyze your data and the search needs of your customers and come up with a good setup of fields, field types, field weights, and the text search specific issues like stemming or multiple languages, handling of special characters, splitting compound words into single words (very important for german search, we can chain words arbitrarily here).

In addition you can do a lot of statistics like counting words or phrases your users search for and compare them against statistics of your index data (for example to find out what synonyms are relevant for you)

Improve this answer
2
  • 1
    Stemming is probably the most important thing that full text search engines do that your typical database usually doesn't do: "check", "checking", checked" and "checker" will be identified as related (or even identical) and searching for one will find the other. That makes a huge difference in the usability of a search.
    – Joachim Sauer
    Commented Jan 24, 2013 at 9:40
  • Some database engines do stemming - the full text indexing in MS SQL Server does that (blogs.msdn.com/b/sqlserverfaq/archive/2009/09/28/… - look for the FORMSOF keyword, I think). WHether it does the best job of it, I don't know.
    – JohnL
    Commented Jan 24, 2013 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.