I need to extract fields like the document number, date, and invoice amount from a bunch of .csv files, which I believe are referred to as "unstructured text." I have some labeled input files and will use the NLTK and Python to design a data extraction algorithm.

For the first round of classification, I plan to use tf-idf weighting with a classifier to identify the document type - there are multiple files that use the same format.

At this point, I need I way to extract the field from the document, given that it is X type of document. I thought about using features like the "most common numbers" or "largest number with a comma" to find the invoice amount, for example, but since the invoice amount can any numerical value I believe the sample size would be smaller than the number of possible features? (I have no training here, bear with me.)

Is there a better way to do the second part? I think the first part should be okay, but I'm not sure that second part will work or if I even really understand the problem. How is my approach in general? I'm new to this kind of thing and this was the best I could come up with.

  • Do you have any knowledge of how many document types there are? If they're csv they have at least some structure, and if they're all different sorts of invoice, the terms in them may not be very informative as predictive features. – Sean Easter Jul 5 '15 at 14:30
  • Yes, this is the case. Which predictive features would you examine? – DataDude Jul 6 '15 at 17:11
  • I can't give a good answer to that without seeing the data :) Feature selection is where much of the art is. In general, you want features that can help you separate the items you're trying to classify. For instance, if there are particular terms in the column headings that correspond to the different categories, then the presence of those words will probably make for useful features. (I can expand on this in an answer if it's helpful.) You may have more luck if you ask this question on stats.stackexchange.com – Sean Easter Jul 9 '15 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.