1

What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a similar question on what language to write a general-purpose ML library in, but not one for data formats. I have a number of requirements for the library in mind, but I do not know how to evaluate a potential data format in regards to these requirements and how to choose which one(s) best fit my use.

Requirements

  • The library is not designed to be used with any other specific program/library in mind, i.e. I don't already "know" beforehand that it will be used by e.g. gnuplot
  • The library is for exploratory research and so it's not being made with a specific "real-life" application in mind
  • The library is intended to be a simple "input-output" data-processing library (cf. the Unix philosophy)
  • At the moment, I will be the primary user of the library but I will more than likely be sharing the data with other people in the future (although I'm not yet sure whom exactly) and I intend to make the library freely available online in some form, so it's hard to say exactly who will be using my data format(s)
  • The amount of data handled is quite large but not astronomical
  • Human readability would be a huge advantage so that people can "eyeball" the data for insight/error checking
  • Performance is not a huge issue since this is not for real-time processing
1
  • Are you asking for a data format that meets these requirements? (which seems off-topic for the same reasons as tool/library recommendations) Or are you asking whether this list of requirements is the correct one for all machine language tasks which require a data format? (which seems both too broad and opinion-based)
    – Ixrec
    Commented Apr 19, 2016 at 8:49

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.