I have seen similar questions about text comparison, but none on such a large scale.
I have a client with two sets of recordings of speeches, 250 and 550 recordings each. Each of the recordings in each set are unique, but about 200 out of the 250 are duplicated in the set of 550, and they need to weed out the doubles. The recordings were recorded on separate devices, so you can't just compare the file length and content.
They are willing to pay to have the first 15 seconds of each recording transcribed, and then they want me to devise some system of using those transcripts to identify the doubles.
First of all, I'm assuming that I should first remove any punctuation and capitalization, because those are things that might arbitrarily change based on the transcriptionist who happens to do that file.
My first idea was to simply calculate the Levenshtein distance for each combination of recordings, but I can't imagine doing that on such a large scale (250 x 550) would be efficient. Then I thought that instead of doing a Levenshtein distance, but using words as the unit instead of characters. But this still would involve running an algorithm 250X550 times, albeit a much faster algorithm each time.
Then I was thinking that maybe I should go through each document, and make a alphabetically sorted dictionary with a count of how many times each word appears in each document. Then I could just go through the dictionary associated with each document and subtract the number of appearances of each word from each other, to produce a total number of mismatched words. This looses taking into account the order of the words, but it should be much faster.
Which of these methods, or perhaps an entirely different method, should I use?