1

What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a similar question on what language to write a general-purpose ML library in, but not one for data formats. I have a number of requirements for the library in mind, but I do not know how to evaluate a potential data format in regards to these requirements and how to choose which one(s) best fit my use.

Requirements

  • The library is not designed to be used with any other specific program/library in mind, i.e. I don't already "know" beforehand that it will be used by e.g. gnuplot
  • The library is for exploratory research and so it's not being made with a specific "real-life" application in mind
  • The library is intended to be a simple "input-output" data-processing library (cf. the Unix philosophy)
  • At the moment, I will be the primary user of the library but I will more than likely be sharing the data with other people in the future (although I'm not yet sure whom exactly) and I intend to make the library freely available online in some form, so it's hard to say exactly who will be using my data format(s)
  • The amount of data handled is quite large but not astronomical
  • Human readability would be a huge advantage so that people can "eyeball" the data for insight/error checking
  • Performance is not a huge issue since this is not for real-time processing
  • Are you asking for a data format that meets these requirements? (which seems off-topic for the same reasons as tool/library recommendations) Or are you asking whether this list of requirements is the correct one for all machine language tasks which require a data format? (which seems both too broad and opinion-based) – Ixrec Apr 19 '16 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.