I have written a program that can rapidly (within 5 sec on a 2GB RAM desktop, 2.33 Ghz CPU) differentiate between structured text (e.g English text) and random alphanumeric strings. It can also provide a probability score for the prediction.

Are there any practical applications/uses of such a program? Note that the program is based on entropy models and does not have any dictionary comparisons in its workflow.

4 Answers 4


You could use it to scan through random text blocks and identify patterns and use these patterns for "predicting" the future. Kinda like an automated Nostradamus. :-)

On a more serious note, perhaps you could use it to select word/phrase-like strings from a bunch of randomly generated strings. These are usually desirable passwords because they are easier to remember and usually wouldn't appear in a dictionary.


The typical application for this (categorizing data based on previous training data - assuming that is indeed what this does) would be for an adaptive spam filter. Other things that I can think of:

  • categorize texts by language
  • estimate entropy (the higher the probability for "completely random", the better your entropy)
  • salvage textual content from damaged files or partitions
  • estimate the quality of forum posts (higher probability for "structured text" means more likely to be properly written)
  • find clear-text payload in sniffed network communications
  • categorize texts by some stylistic property, e.g. formality level (slang / informal / formal / ...) or poetic vs. factual (is this bit of text from a novel or from a news article?)

Whether your algorithm is suitable for these problems depends, but if it does what I think it does, you should be able to apply it to any of them.


It could be used as part of a program like the Unix file command to distinguish "English text" from unstructured "data".


You could "reverse" the algorithm, to generate natural-looking phrases. This could be used in games to generate the text on signs and posters, newspapers on the floor, etc. in a procedurally generated world. The text would look intelligible, but still look like natural language.

If I understand your algorithm correctly, you do some kind of measure on entropy, so it won't accurately model which letters should be used by the text. If that is the case, a simple tweak to generate likely alternations of vowels and consonants, like proposed here : http://www.umbrarumregnum.net/articles/creating-names.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.