We are looking for a way to tokenize some text in the same or similar way as a search engine would do it.

The reason we are doing this is so that we can run some statistical analysis on the tokens. The language we are using is python, so would prefer a technique that works in that language, but could probably set something up to use another language if necessary.


Original token:

We have some great burritos!

More simplified: (remove plurals and punctuation)

We have some great burrito

Even more simplified: (remove superfluous words)

great burrito

Best: (recognize positive and negative meaning):

burrito -positive-
  • Look into parser combinators, here's a parser combinator library for python valuedlessons.com/2008/02/… and another here pyparsing.wikispaces.com parser combinators allow for almost DSL like approach to parsing based on match attempts similar to regexp but far more flexible/powerful Commented Nov 15, 2012 at 20:22
  • The other approach that comes to mind (but feels a little..funny...) is to simply write a token database where you have the unique tokens you want in one table, and a blacklist table for tokens to strip and a white list table for dupe tokens where you have an ID that maps many tokens to one of the token records in the tokens you want table. Basically a dictionary database that translates between the two languages you want where one of the languages is just the minimized vocabulary that "great" "awesome" "terrific" all translate to the same thing in "-positive-" Commented Nov 15, 2012 at 20:25

1 Answer 1


Python has a great natural language toolkit, the NLTK. It supports word tokenisation out of the box:

>>> import nltk
>>> input = 'We have some great burritos!'
>>> tokens = nltk.word_tokenize(input)
>>> tokens
['We', 'have', 'some', 'great', 'burritos', '!']
>>> nltk.pos_tag(tokens)
[('We', 'PRP'), ('have', 'VBP'), ('some', 'DT'), ('great', 'JJ'), ('burritos', 'NNS'), ('!', '.')]

The last structure includes natural language tags, which allow you to drop words from consideration based on their classification. You probably want to focus on the JJ (adjective) and NN-prefixed (noun) tags.

From there on out you can apply stemming, and detect positive and negative adjectives.

I believe that for adjective classification, however, you'd need to create your own corpus from online resources such as this; the library does give you the tools for this.

Here is a stemming example using the Porter stemming algorithm:

>>> from nltk.stem.porter import PorterStemmer
>>> PorterStemmer().stem('burritos')

O'Reilly published a book on the library, now available online.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.