What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a similar question on what language to write a general-purpose ML library in, but not one for data formats. I have a number of requirements for the library in mind, but I do not know how to evaluate a potential data format in regards to these requirements and how to choose which one(s) best fit my use.
- The library is not designed to be used with any other specific program/library in mind, i.e. I don't already "know" beforehand that it will be used by e.g. gnuplot
- The library is for exploratory research and so it's not being made with a specific "real-life" application in mind
- The library is intended to be a simple "input-output" data-processing library (cf. the Unix philosophy)
- At the moment, I will be the primary user of the library but I will more than likely be sharing the data with other people in the future (although I'm not yet sure whom exactly) and I intend to make the library freely available online in some form, so it's hard to say exactly who will be using my data format(s)
- The amount of data handled is quite large but not astronomical
- Human readability would be a huge advantage so that people can "eyeball" the data for insight/error checking
- Performance is not a huge issue since this is not for real-time processing