I am trying to write a spellchecker, I have a huge wordlist (at least 500K, because of the nature of the language). The performance would suffer a lot if I got the lavenshtein distance of all of the words in the wordlist to the current word.
So I am trying to distribute the wordlist into buckets based on the characters in the word. This way, I only need to get the Levenshtein distance of the words in the specific bucket instead of the whole wordlist.
So I was wondering if there are any good Locality Sensitive hash algorithms for my problem? I have written a very simple algorithm that seems to work okay and the performance is at least 20X better. Although the accuracy suffers quite a bit.
Here is what I have come up with:
static string GetHash(string word)
{
word = word.ToUpperInvariant();
var lettersToTake = word.Length - (int)(word.Length * 0.70);
var chars = word.GroupBy(c => c)
.OrderByDescending(c => c.Count())
.ThenBy(c => GetFrequency(c.Key))
.Take(lettersToTake)
.Select(g => g.Key)
.ToList();
return new string(chars.ToArray());
}
Here are the steps of the algorithm:
- Capitalize all of the characters
- Order the characters in the word by the frequency of the letter in the word
- Then order the characters by the frequency of the word in the entire wordlist (GetFrequency)
- Take 30% of the characters in the word
adres
should be corrected toaddress
and notacres
, even thoughacres
has a shorter edit distance.