0

I found this Python code implementing Double Metaphone, and it basically has a bunch of if statements handling each letter, though I'm not exactly sure the decisions that went into each branch yet. But the output are two hashes of consonants it seems.

result = doublemetaphone('Jose')
self.assertEquals(result, ('HS', ''))
result = doublemetaphone('cambrillo')
self.assertEquals(result, ('KMPRL', 'KMPR'))
result = doublemetaphone('otto')
self.assertEquals(result, ('AT', ''))
result = doublemetaphone('aubrey')
self.assertEquals(result, ('APR', ''))
result = doublemetaphone('maurice')
self.assertEquals(result, ('MRS', ''))
result = doublemetaphone('auto')
self.assertEquals(result, ('AT', ''))
result = doublemetaphone('maisey')
self.assertEquals(result, ('MS', ''))
result = doublemetaphone('catherine')
self.assertEquals(result, ('K0RN', 'KTRN'))

The question I have now is, what are you supposed to do with these hashes? Say I have 1 million words and I convert them all to hashes, do I put them into some sort of trie, and just do a basic trie lookup check somehow? If so, roughly what goes on there? If not, what is actually done with the hashes to make the Metaphone algorithm work on large dictionary datasets?

The use-case I am trying to figure out is basically, how do I implement a nice search feature for a multi-language dictionary (dictionaries for each of several individual isolated languages: Chinese, Sanskrit, Hebrew, Arabic, etc.)? It sounds like using the Metaphone-like approach, you would convert input queries to a pseudo-phonetic hash, then I'm just not sure what you would then do with those hashes basically, to implement fuzzy search to find close matching dictionary terms.

1 Answer 1

2

You didn't supply a use case, so I will make one up for you.

def get_metaphone_to_word(in_file="product_catalog.txt"):
    m_to_word = {}
    with open(infile) as fin:
        for line in fin:
            for word in line.lower().split():
                m1, m2 = metaphone(word)
                m_to_word[m1] = word
                m_to_word[m2] = word
    return m_to_word

m_to_word = get_metaphone_to_word()

def fix_product_typos(line):
    out_words = []
    for word in line.lower().split():
        m1, m2 = metaphone(word)
        w1 = m_to_word.get(m1)
        w2 = m_to_word.get(m2)
        if w2:
            out_words.append(w2)
        elif w1:
            out_words.append(w1)
        else:
            # We were unable to cleanup a possibly misspelled word.
            # It might be an English word or surname not in our catalog.
            out_words.append(word)

    return " ".join(out_words)

Apply fix_product_typos() to each line of Slack, email, or other customer input, to coerce the input terms toward a restricted vocabulary, such as widget names found in a product catalog.

Alternatively, one might feed it /usr/share/dict/words in hopes of correcting English language typos.

There's a "last update wins!" aspect to hash collisions, so it will work better with an input file that is sorted, so rare terms appear first, with popular terms near the end.


result = doublemetaphone('cambrillo')
self.assertEquals(result, ('KMPRL', 'KMPR'))

nit: Better to phrase each AAA as a one-liner for these simple tests.

self.assertEquals(doublemetaphone('cambrillo'), ('KMPRL', 'KMPR'))

Nothing wrong with repeated copy-n-paste for a simple test suite like this. But you might also consider iterating through a list:

    for word, expected_metaphones in [
        ...
        ('cambrillo'), ('KMPRL', 'KMPR'),
        ...
    ]:
        self.assertEquals(doublemetaphone(word), expected)
6
  • So is it fair to say, if you were to use this for autocomplete or spell check, you would convert the input word to metaphone hash, then fetch all the words which have that same hash? Is it that easy? Doesn't that cause false matches is what I'm wondering next. And you have to have clean input data to start, to do your "cleanup" of product data it looks like in your example, so that's a barrier to entry I guess. Updated the question with my basic use-case pretty much.
    – Lance
    Nov 2, 2023 at 2:33
  • Yes, and yes. The code I offered mostly ignores the collision aspect due to the "last one wins!" overwriting. Certainly one could maintain a list of colliding words. Prolly you would want to augment that with word frequencies, leading to a "most frequent word wins!" rule. // I have a hard time seeing how "autocomplete" is a good use case for Metaphone or similar Soundex schemes. You kind of need to wait for 80% of a word's characters to be entered before you can usefully probe the dict. Autocorrect spell check? Yeah, maybe that would work OK. An indexed RDBMS table would be a good fit.
    – J_H
    Nov 2, 2023 at 2:37
  • 1
    Ummm, I'm looking at your revised Question, which mentions several languages. Wow, those are hard languages! In the sense that I don't see a bunch of {Germanic, Romance, Slavic} tongues. You mentioned a pair of BIDI languages which read right-to-left, which I guess is not a deal breaker. Ideograms don't fit into the whole Soundex-and-derivatives at all. Sanskrit seems like it could plausibly work. But you might be happier with word2vec embeddings. Maybe a "sounds alike" typo homophone kind of embedding? Then the Sanskrit, Hebrew, and Chinese words for "dog" would be near, in embedding space.
    – J_H
    Nov 2, 2023 at 2:50
  • Interesting, will have to think about what you're suggesting more. Digging more into the metaphone algorithm, I'm basically now wondering why the Double Metaphone algorithm chose to substitute and merge consonants, so been digging into that but not finding much.
    – Lance
    Nov 2, 2023 at 3:12
  • 1
    Oh, sorry, I meant to say this before running out of comment characters. The various Soundex derivatives are pretty much language specific, tuned for regional use cases. That is, they pay attention to what would be a homophone in the target language. So I'm skeptical that they would be a good fit for your multi-language use case, especially since you target languages from tree branches that are pretty far apart from one another.
    – J_H
    Nov 2, 2023 at 3:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.