A very useful learning tool I stumbled across for Chinese was a massive list of sentences that, barring the first 10 or 15, only differed by the ones before by one or two words, or at least as few as possible: The creator had sorted them by relation, obviously starting with a few standard sentences to prevent the first ones from being random and complex.
I am a little confused about how to go about writing my own implementation of such and algorithm. Now bear with me (unless you stopped reading b/c this question is out of line for this site) as I have been called "stupid" by several teachers but I was thinking of this incredibly inefficient algorithm:
- Take first sentence (sentence number one)
- Cycle through each sentence thereafter and count the number of different words (different from 1st sentence) (by looping through each word of each sentence) - assign a number in an array depicting this.
- Cycle again and find those with the lowest associated numbers
- If there is one sentence with the lowest number of different words - make that the next sentence in the list (sentence number two)
- If there are multiple sentences, sort them by which have the most words in the top 1000 words in the language and thus are most useful
- With the selected sentences (with the most similar words to sentence 1) sorted by word frequency, choose the one or ones that are most frequent as sentences 2, 3 etc.
- Start all over by checking the next sentence in the sequence against the last one combined with all previous ones to find lowest number of new words across all sentences
For a corpus of 100, 000 sentences, I will be looping millions of times for each few sentences most likely. There must be a better way. My overall goal is as follows:
Have each sentence in the array (sequence, what have you) have as few new words as possible, with "new" words defined as those not present in any previous sentences.