0

Considering that using a relational DB or other hierarchical or such databases is not an option, looking for the alternative approach using a simple (but very fast, efficient, distributed KV-store).

The value (referred by key) is an encrypted BLOB, whose contents are 2 related entities / objects, with a 1:N 'refers-to' relationship. E.g.

'Master' ==> (0:N) 'Pet'

Where Master has many attributes (Unique-Membership-Id, Name, Address, DoB, Gender...), Pet similarly has many attributes (Name, DoB, current-master-Unique-Membership-Id, genus, class, gender...), and pet-ownership may change (just an example).

In such a case would XML, or JSON or another encoding format would be most efficient (from read/decode/compute-speed standpoint, not from storage efficiency standpoint), assuming that a Java application needs to read/decode this information very fast (compute latency needs to be least). Writes are infrequent and latency of same isn't an issue. Also, note that I'd use KV datastore to also hold indices to allow search/lookup using secondary keys. I'm aware of the needs to maintain consistency and referential integrity, explicitly in my application, for update operations.

Based on my research (reading existing SO/Prog SE and other sources), JSON seems to be the clear winner for straight-foward non-relational data. Is there something that is even faster ? Does the fact that my data is relational, though relatively simple, impact choice between JSON, XML or another form ?

  • 1
    Fixed width fiels will allow you to jump to record x – Ewan Apr 25 '16 at 7:33
  • Storage efficient scheme could be faster, i.e. compressed data take less i/o time. – imel96 Apr 25 '16 at 9:28
  • Compression (& encryption) is an existing design choice which I don't have influence over (or, might be very difficult to change). Thanks for the suggestion. – icarus74 Apr 27 '16 at 12:57
2

You can do better than either XML or JSON by exploiting the particular characteristics of your particular data. If the data is completely flat and every row contains the same fields, then CSV is going to be more efficient than either XML or JSON. The reason people generally prefer formats like XML and JSON is because they recognise that the life-time cost of an application and the life-time value of the data is more important than raw machine performance, which is about the cheapest resource in your entire system.

  • +1. I would like to add that, even if his data is flat right now, in further upgrades where data could end up being complex, to use CSV would force to him to recode the stage of key/value mapping. I'm in agreement with @Michael Kay on using CSV but, If It depends on me, I would prefer JSON (lighter than xml) becaus It would be ready for complex data models. It's also the meta-data used in no-Sql DB like Mongo or Redis. To search in JSON data structures is easier than to search in CSV. I like to be cautious – Laiv Apr 25 '16 at 9:32
1

The most efficient would probably be a binary format which you could read directly into memory, skipping the parsing step.

  • Most of the time you will convert the data into some kind of object, and that is likely more time consuming than parsing JSON. – gnasher729 Apr 25 '16 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.