Product classifying algorithm - text classification - C# - algorithm suggestions

Question

Alright people. Finally with the help of stackoverflow community i have gathered 20 commercial product selling websites product pages with the following features

Product URL
Product Price
Product Name
Product Category
Product Page Title
Product Page Description
Product Page Keywords

Now with using these features of the products i have to classify them. What does classification means ? Let me explain it.

Now as you can imagine every website lists the products in their own ways. There is not format. So lets say iphone 4 is being sold at 20 different websites with 20 different way. So what i need to achieve is grouping the these 20 iphone page at 20 different websites. So when person query my website with iphone 4 word i will show that 20 results. Basically out of over 500.000 product urls i need to group every product. So lets say there are 15 gefore gtx 570 card out of these 500k urls so i need to group them as same product. You can imagine it as google products. But i am doing it at my own country which is Turkey and google does not have product search for Turkey.

In short with using the features above what algorithm would you suggest. I don't want to use any training techniques if possible. Everything automated.

I am using C# 4.0 WPF and data is stored at the MSSQL 2008 R2 database

Answer 1

Wouldn't you just need to reverse index the words in the various fields back to the page? As a simplistic example, break down each Product Name (say, by spaces) and normalize each keyword (say, lowercase, throw out punctuation), and then map it to the product's URL or record. Searching for "iphone" would then give you close to the right results. You even have a Product Page Keywords field.

Product grouping would then just be a search on a combination of keywords, like "iphone case".

Getting such a system perfect, of course, can be immensely complex, but the basic logic seems like it would apply here. For example, your tokenization of Turkish terms may require special treatment.

Answer 2

I would realize that features as Decorators from the Decorators pattern. Ie, each could envelope another decorator. So, the every real object will have features : FirstObject, FirstObject.envelopes, FirstObject.envelopes.envelopes, ... up to the last. You will have to make a function, that shows for any decorators all inner ones. And the function that will make from your DB such "MATRYOSHKA".

Any decorator will have its specific fields:

1. name of the feature
2. Value of the feature. A list of possible feature values should be set as a static list into every Decorator class.
3. Where it would be shown on what page at which format. This I would realize as a map of page types as keys and values=objects of a class Placement, will include place and format. These placements will do the real showing of the feature on the page.
4. Will you also analyze other pages? For here you could set also places where on the pages these features could be found. But this is very hard point.
5. Here you could put dependencies - what decorators could envelop by which.
6. Also here should be a mechanism for passing important information from the inner decorators to the outer ones. But that is the part of the design pattern standard.

The other variant - every object is a list of such features, only independent ones. But that is, IMHO, too dynamic construction for your task, and is harder to control.

This is the most primitive model.

As for more elaborated ones, you should take into account synonyms of the feature names. That means, that the list of possible feature values will change to list of lists.

The next step - some features could exist apart or together as one string. For example, GeForce or 590 or GeForce 590. For realization of this you should use Congregator instead of Decorator.

One more step - you should consider features that are represented by numbers (price, date). It is independent on the two previous steps. Instead of lists of possible values, the decorator will have a range of possible values. The search pattern will be changed so, that the range setting will be possible.

The next independent step - ordering by features. You have all features ordered by including and all values are ordered by order in the lists of possible values. This is the natural order. But you could change the order.

So, two power instruments you should do: searcher UI and orderer UI. IMHO, try not to do the search or ordering for these objects by your code, use SQL instead. Objects should only represent them, used for input and output and setting search/ordering UI definition.

Of course, it could be elaborated further, but it would be better to realize the task step by step.