How would you go about parsing a sentence like "Bought two kilos of steak from Acme supermarket" into a data structure like the following JSON representation:

  item: {name: "steak", tags: "meat,beef" },
  quantity: { value: 2, unit: "kg"},
  source: "ACME Supermarket"


I'm looking for a high-level conceptual overview of where to start e.g. some papers or introductory material that don't require PhD level knowledge. For context, this is part of the preliminary investigation for a personal expense tracker I'm planning to build.

To break it down a little further, I'm interested in basic named entity recognition, and categorization strategies. I don't have a CS education, so you may want to keep that in mind when answering :) Thanks in advance. Not interested in 3rd party web services, since this is a learning exercise, and is intended to work offline.

closed as too broad by user40980, Ixrec, GlenH7, Kilian Foth, Bart van Ingen Schenau Feb 16 '16 at 12:17

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    I'm sure you're aware but natural language processing is no simple task. Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo. – Mike Dec 12 '13 at 20:14
  • Ha! Yes I am, and painfully so. I think I've seen that before, possibly from nlpwp.org or the early material from pre-Coursera nlp-class.org. I am, however, hoping the reduced problem space makes for "simpler", if not necessarily "simple", approaches. – Okal Otieno Dec 12 '13 at 20:19
  • 2
    PhD level knowledge doesn't include how to do this. It's an extraordinarily complicated question...Imagine the amount of work that would go into categorizing a simple list of things that can be bought. Now, imagine a category list of places that sell. Now build all the natural language causal links between the two. Now add another person. – Satanicpuppy Dec 12 '13 at 20:20
  • What I'm hoping for is something just slightly more sophisticated than looking for synonyms of "bought" and determining the object of the verb. I worked on a retail ordering system that accepted that sort of input, but it was pretty much a spelling corrector coupled with a dictionary that failed if it couldn't find a reasonable match. The solution needn't be probablistic. I realize my optimism is likely based on extreme naivete, but surely, there's got to be somewhere to start? :( – Okal Otieno Dec 12 '13 at 20:34
  • If the problem space is sufficiently simplified, you're looking at nothing more than a Domain-Specific Language (DSL). – Robert Harvey Dec 12 '13 at 20:49

Your best bet is to define your own syntax which appears to be natural (or close-to-natural) language, even though it's actually much more rigid.

For example, a SQL select statement: SELECT <something> AS <foo> FROM <table> WHERE <something else> IS <a value> (note that I replaced = with IS to make the point, but that the capitalization is not important).

In your case, it would be BOUGHT <quantity> OF <thing> FROM <location>. Then you just have to parse <quantity>, <thing>, and <location> to match them up against known items.

switch (quantityString)
   case "a": 
   case "1": 
   case "a single": 
   case "one":
     return 1;
   case "a pair":
   case "2":
   case "two":
     return 2;

You could even make the first word variable input too, but only from a select set of known verbs (the way SQL has SELECT, DELETE, INSERT, etc). You could handle derivatives of them ("purchased" vs "bought") similarly.

This would be an example of defining a Domain-Specific Language like @Robert-Harvey mentioned.

  • Thanks, I just might go with this approach. Can't upvote you until I hit 15 though, sorry :( I was hoping for something between this and Siri, but it increasingly looks like there's no middle ground. Might go with a thesaurus to make it a little more "intelligent". – Okal Otieno Dec 12 '13 at 21:37
  • @OkalOtieno - Yeah, a thesaurus could certainly work, if you can implement it to have reasonably quick lookups. Just be aware that tenses can really mess it up. Buy -> Bought vs Purchase -> Purchased. – Bobson Dec 12 '13 at 21:43
  • 1
    The trick is to have people who don't know how your program works provide test phrases. – Brian Dec 12 '13 at 22:19
  • Sounds like a great idea. I'll get my co-workers to give me sample inputs. Thank you :) – Okal Otieno Dec 13 '13 at 7:25
  • 1
    @Bobson Thanks for taking time to respond :) I'm still holding out hope for different suggestions. Shall accept it once it's clear that this is the best strategy, which seems to be the case at the moment. – Okal Otieno Dec 13 '13 at 21:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.