I am currently building a small in-house search engine based on Apache Lucene. Its purpose is simple - based on some keywords, it will suggest some articles written internally within our company. I am using a fairly standard TF-IDF scoring as a base metric and built my own scoring mechanism on top it. All of these seem to be working excellent except for some corner cases where the ranking seems messed up.
So what I am planning on doing is to add a small Relevant/Not Relevant link to the search results page so that users can click on one of those depending on their perception of whether that result should have been included in the first place.
My Idea
- Treat these Relevant/Not Relevant as labels and create a training data.
- Use this data to train a classifier (such as SVM)
- Incorporate this model into the search engine i.e., every new result will pass through the classifier and will be assigned a label on whether it is relevant or not.
This approach seems intuitive to me but am not sure whether it will work in practice. I have two specific questions:
- What all features should I extract?
- Is there a better way to integrate the machine learning component into the search engine? My final goal is to "learn" the ranking function based on both business logic as well as user feedback.