How would you go about parsing a sentence like "Bought two kilos of steak from Acme supermarket" into a data structure like the following JSON representation:

  item: {name: "steak", tags: "meat,beef" },
  quantity: { value: 2, unit: "kg"},
  source: "ACME Supermarket"


I'm looking for a high-level conceptual overview of where to start e.g. some papers or introductory material that don't require PhD level knowledge. For context, this is part of the preliminary investigation for a personal expense tracker I'm planning to build.

To break it down a little further, I'm interested in basic named entity recognition, and categorization strategies. I don't have a CS education, so you may want to keep that in mind when answering :) Thanks in advance. Not interested in 3rd party web services, since this is a learning exercise, and is intended to work offline.

  • 5
    I'm sure you're aware but natural language processing is no simple task. Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo. – Mike Dec 12 '13 at 20:14
  • Ha! Yes I am, and painfully so. I think I've seen that before, possibly from nlpwp.org or the early material from pre-Coursera nlp-class.org. I am, however, hoping the reduced problem space makes for "simpler", if not necessarily "simple", approaches. – Okal Otieno Dec 12 '13 at 20:19
  • 2
    PhD level knowledge doesn't include how to do this. It's an extraordinarily complicated question...Imagine the amount of work that would go into categorizing a simple list of things that can be bought. Now, imagine a category list of places that sell. Now build all the natural language causal links between the two. Now add another person. – Satanicpuppy Dec 12 '13 at 20:20
  • What I'm hoping for is something just slightly more sophisticated than looking for synonyms of "bought" and determining the object of the verb. I worked on a retail ordering system that accepted that sort of input, but it was pretty much a spelling corrector coupled with a dictionary that failed if it couldn't find a reasonable match. The solution needn't be probablistic. I realize my optimism is likely based on extreme naivete, but surely, there's got to be somewhere to start? :( – Okal Otieno Dec 12 '13 at 20:34
  • If the problem space is sufficiently simplified, you're looking at nothing more than a Domain-Specific Language (DSL). – Robert Harvey Dec 12 '13 at 20:49

Your best bet is to define your own syntax which appears to be natural (or close-to-natural) language, even though it's actually much more rigid.

For example, a SQL select statement: SELECT <something> AS <foo> FROM <table> WHERE <something else> IS <a value> (note that I replaced = with IS to make the point, but that the capitalization is not important).

In your case, it would be BOUGHT <quantity> OF <thing> FROM <location>. Then you just have to parse <quantity>, <thing>, and <location> to match them up against known items.

switch (quantityString)
   case "a": 
   case "1": 
   case "a single": 
   case "one":
     return 1;
   case "a pair":
   case "2":
   case "two":
     return 2;

You could even make the first word variable input too, but only from a select set of known verbs (the way SQL has SELECT, DELETE, INSERT, etc). You could handle derivatives of them ("purchased" vs "bought") similarly.

This would be an example of defining a Domain-Specific Language like @Robert-Harvey mentioned.

| improve this answer | |
  • Thanks, I just might go with this approach. Can't upvote you until I hit 15 though, sorry :( I was hoping for something between this and Siri, but it increasingly looks like there's no middle ground. Might go with a thesaurus to make it a little more "intelligent". – Okal Otieno Dec 12 '13 at 21:37
  • @OkalOtieno - Yeah, a thesaurus could certainly work, if you can implement it to have reasonably quick lookups. Just be aware that tenses can really mess it up. Buy -> Bought vs Purchase -> Purchased. – Bobson Dec 12 '13 at 21:43
  • 1
    The trick is to have people who don't know how your program works provide test phrases. – Brian Dec 12 '13 at 22:19
  • Sounds like a great idea. I'll get my co-workers to give me sample inputs. Thank you :) – Okal Otieno Dec 13 '13 at 7:25
  • 1
    @Bobson Thanks for taking time to respond :) I'm still holding out hope for different suggestions. Shall accept it once it's clear that this is the best strategy, which seems to be the case at the moment. – Okal Otieno Dec 13 '13 at 21:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.