2

I was told to create two lists concerning most frequently used words from a plain text (10 Mb arbitrary texts) as monograms (for single worded expressions such as human, water, is) and bigrams (for two-worded expressions such as basketball team, united states and etc)

I am stuck here and don't know how I can go about it! And how I can distinguish between these two?

My domain is not English, I only gave those examples to make my intention and meaning more clear.

Improve this question
9
  • 2
    You may be over-thinking it... a 'bigram' could simply be any pair of adjacent words, not necessarily a related pair. So in your post, some example bigrams might be "My domain", "domain is", "is not", and "not English". You should probably get clarification.
    – Hellion
    Commented Jul 15, 2013 at 16:02
  • 1
    @Hellion there are certain word pairs that have a higher occurrence than random, that might be what is desired... though yes, further clarification is a good idea.
    – user40980
    Commented Jul 15, 2013 at 16:07
  • Thank you guys, So if we assume those word pairs that have meanings are desired how should i go about it? In first scenario where i can choose any two adjacent words as a pair, i would definitely face problems since i would easily make a meaningful pair into a nonsense ! suppose: I am a basketbal team player that play in UNITED STATES national team I am a basketbal team player that play in UNited States national team! See its really hard to understand where to start to how to go about it.:-/
    – Hossein
    Commented Jul 15, 2013 at 16:13
  • 2
    You might be headed in the direction of the markov chain - specifically, the text generator (or at least creating the structure to run one).
    – user40980
    Commented Jul 15, 2013 at 16:18
  • 1
    In NLP, "bigrams" are always two adjacent words in an input stream. So counting frequent bigrams is essentially the same as counting frequent words: keep track of how often each pair of adjacent words occurs, and return the ones with the highest count - the only problem is that storing pair counts requires much more memory.
    – Kilian Foth
    Commented Jul 15, 2013 at 17:40

1 Answer 1

Reset to default
1

You can try to read from text word by word and make 2 instances of Dictionary, one for monograms and one for bigrams, having the expression as Key and occurrence as Value. With this you can make some statistics about expressions usage. You can also use database storage for bigger files.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.