Possible applications of algorithm devised for differentiating between structured vs random text

Question

I have written a program that can rapidly (within 5 sec on a 2GB RAM desktop, 2.33 Ghz CPU) differentiate between structured text (e.g English text) and random alphanumeric strings. It can also provide a probability score for the prediction.

Are there any practical applications/uses of such a program? Note that the program is based on entropy models and does not have any dictionary comparisons in its workflow.

Answer 1

You could use it to scan through random text blocks and identify patterns and use these patterns for "predicting" the future. Kinda like an automated Nostradamus. :-)

On a more serious note, perhaps you could use it to select word/phrase-like strings from a bunch of randomly generated strings. These are usually desirable passwords because they are easier to remember and usually wouldn't appear in a dictionary.

Answer 2

The typical application for this (categorizing data based on previous training data - assuming that is indeed what this does) would be for an adaptive spam filter. Other things that I can think of:

• categorize texts by language
• estimate entropy (the higher the probability for "completely random", the better your entropy)
• salvage textual content from damaged files or partitions
• estimate the quality of forum posts (higher probability for "structured text" means more likely to be properly written)
• find clear-text payload in sniffed network communications
• categorize texts by some stylistic property, e.g. formality level (slang / informal / formal / ...) or poetic vs. factual (is this bit of text from a novel or from a news article?)

Whether your algorithm is suitable for these problems depends, but if it does what I think it does, you should be able to apply it to any of them.

Answer 3

It could be used as part of a program like the Unix file command to distinguish "English text" from unstructured "data".

Answer 4

You could "reverse" the algorithm, to generate natural-looking phrases. This could be used in games to generate the text on signs and posters, newspapers on the floor, etc. in a procedurally generated world. The text would look intelligible, but still look like natural language.

If I understand your algorithm correctly, you do some kind of measure on entropy, so it won't accurately model which letters should be used by the text. If that is the case, a simple tweak to generate likely alternations of vowels and consonants, like proposed here : http://www.umbrarumregnum.net/articles/creating-names.