I have a website that allows admin users to input free text questions to a database that will end up being asked to non-admin users using the website. This system is replacing an older system in which the library of questions became almost useless due to the amount of duplicate questions so one of my main goals is to do my best to detect and force the reuse of existing questions.
What strategies do I have to avoid ending up with the same issue? My initial thought is to compare a user input to existing data using a search engine (CloudSearch, Elastic Search, etc) and blocking question input on a certain score threshold.
Are there other strategies? Is there any specific way I should think about storing/sanitizing my data to maximize efficiency? I'm thinking of things like removing low value words (the, of, is, etc) to boil down to the essence of the question.
While I am asking this question it seems very meta as perhaps what I am trying to create is the "Similar questions" widget here on Stack Exchange.