0

My node.js app has to deal with (sometimes large; a few MB) tables of numerical data. I'm about to use JavaScript's native binary objects for this purpose (i.e. ArrayBuffer and friends), because they're fast and well-supported by fantastic libraries like scijs-ndarray.

I'm wondering now how to store this data, given that it will have to be manipulated frequently – change a few columns of numbers here, remove and add some rows there, etc.

If I use a blob object in some database, then persisting the changes on a byte-by-byte basis ranges from difficult to impossible.

If I store every Buffer in a binary file, and use filesystem seek/read/write ops, I get finegrained control but I lose convenience, and I don't know what I'd lose in terms of speed, safety and size (maybe it makes sense to run RLE compression on sparse data).

Maybe there are other options (Protobuf? HDF5?) but I don't know whether they really are valid options or if I just haven't done my research. Also, serializing back and forth between formats takes time.

I've never done this kind of thing and completely lack the intuition for what's significant and what's negligible. How would you approach this problem?

  • It rather sounds like your "it's just a buffer of numbers" is quickly turning into a "it's actually a database" sort of situation rather quickly - you're asking to add/remove rows and columns etc - all very database-y sorts of things to do. How would you structure the data if each table was actually a database? Can you combine tables across data samples because the schema is the same by any chance? – J Trana Oct 27 '15 at 6:32
  • They all look totally different, unfortunately; they don't even have the same number of dimensions. I also don't need to make queries - I only read the whole thing at the beginning, and then write small changes to the data later. After a bit more research it looks to me like HDF5 is the way to go, but maybe you have another suggestion? – sebastian_k Oct 27 '15 at 14:11
  • Unfortunately, probably not. The big thing is that "write small changes" can mean either "in-place binary updates or appends" or "randomly insert something in the middle" - one may or may not lead to databases and the other one likely does for sure. As long as you've got that angle covered, you're probably fine. Also, I don't know about your read/write scenarios/scale, and those make a huge difference. – J Trana Oct 28 '15 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.