2

I am new to web development and trying to study various types of websites as case study. Right now my focus is on how search engines works for an eCommerce website.

I know basic functioning for a search engine, i.e. crawl web pages, index them and the display the results using those indexes.

But I got little confuse in case of an eCommerce website. Don't you think that it would be better if a search engine instead of crawling the web pages containing products, it should directly crawl the database and index the products stored in the database? And when a user search for any product, it will simply give us the rows of the table which matches the user query?

If this is not the case, can someone please explain how the usual method works on eCommerce website?

|improve this question
2

Generally, a website like an e-commerce website, and usually also news websites, blogs and anything else with highly structured pages (such as Stack Overflow itself -- though don't try looking for it), will publish an XML sitemap file.

An XML sitemap is basically just a big XML file with a bunch of links to all of the pages in the site. It usually also includes the relative "importance" of each page within the site (so a product page is more important than a category listing, for example) which helps the search engine to decide what to prefer to display in a result page.

Because the sitemap.xml file may be really big (several megabytes for a large site), Google and other search engines also accept more specialized sitemap files. For example, a "News Sitemap" will contains links to all news stories published by a news website in the last 48 hours. Google will "ping" that file much more frequently (even up to once every couple of minutes) to check whether new stories have been published -- this is how they keep http://google.com/news up-to-date.

Note that the sitemap file only contains links to pages within the site. It's not a dump of the database or anything like that. This is because search engines are specifically interested in indexing a site the same way that human would read it. That's why they put so much emphasis on penalizing you from "tricking" it into indexing something a human would never actually see (hidden text and whatnot).

Edit

In response to your question, for the "internal" search functionality, yes most of these will be implemented using a direct search of the database, rather than the typical "public" search engine technique of crawl/index/query.

To make the searching faster, typically something like Lucene is used (some databases provide in-built full-text search functionality as well, but Lucene is more full-featured than any in-built full-text search I've seen). Basically, what comes out of a Lucene search will be the identifier of the products/blog post/news item your query matched, and then you'll fetch those results from the database to display however you like.

However, that can be fairly complex to set up. Many sites will just use something like a Google Custom Search, which basically just does a regular Google search with "site:example.com" pre-pended to the query string. It usually returns worse results than Lucene, but it's pretty trivial to set up.

|improve this answer
  • but here you are talking about the third party search engines like google, bing etc ... but as mentioned in the Title of the question... i am more interested in the search engine/plugin available on the site itself ... like the one we have at the top right corner of this page. – Ravi Gupta Nov 21 '11 at 10:35
  • @RaviGupta: Oh right, I've updated my answer to reflect that as well. – Dean Harding Nov 21 '11 at 11:37
  • Thanks!! ... I looked at the full text search feature available in the mysql, but it is only available with the MyISAM engine... which don't support some of the basic features of RDBMS like transactions, foreign key etc. So should one go for MyISAM or Lucene? – Ravi Gupta Nov 22 '11 at 6:27
  • One possibility would be to denormalize your table and just store the text blobs in a MyISAM table, but everything else in a separate InnoDB table. But personally, I'd go with Lucene - it'll be a little more work to set up, but the additional flexibility will pay for itself. – Dean Harding Nov 22 '11 at 7:36
  • Native Lucene is available in Java so I have two options(as I am working in PHP) i.e. either use a intermediate Java layer or go for the Lucene port for PHP by Zend. But as per stackoverflow.com/questions/5159892/… Zend Lucene is a bad option but instead SOLR is better. What do you suggest? Also at many places people have mentioned about Sphider(sphider.eu), in context of full text search but by looking at its website it seems to me a normal search engine like google? – Ravi Gupta Nov 22 '11 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.