0

Considering that using a relational DB or other hierarchical or such databases is not an option, looking for the alternative approach using a simple (but very fast, efficient, distributed KV-store).

The value (referred by key) is an encrypted BLOB, whose contents are 2 related entities / objects, with a 1:N 'refers-to' relationship. E.g.

'Master' ==> (0:N) 'Pet'

Where Master has many attributes (Unique-Membership-Id, Name, Address, DoB, Gender...), Pet similarly has many attributes (Name, DoB, current-master-Unique-Membership-Id, genus, class, gender...), and pet-ownership may change (just an example).

In such a case would XML, or JSON or another encoding format would be most efficient (from read/decode/compute-speed standpoint, not from storage efficiency standpoint), assuming that a Java application needs to read/decode this information very fast (compute latency needs to be least). Writes are infrequent and latency of same isn't an issue. Also, note that I'd use KV datastore to also hold indices to allow search/lookup using secondary keys. I'm aware of the needs to maintain consistency and referential integrity, explicitly in my application, for update operations.

Based on my research (reading existing SO/Prog SE and other sources), JSON seems to be the clear winner for straight-foward non-relational data. Is there something that is even faster ? Does the fact that my data is relational, though relatively simple, impact choice between JSON, XML or another form ?

Improve this question
3
  • 2
    Fixed width fiels will allow you to jump to record x
    – Ewan
    Commented Apr 25, 2016 at 7:33
  • Storage efficient scheme could be faster, i.e. compressed data take less i/o time.
    – imel96
    Commented Apr 25, 2016 at 9:28
  • Compression (& encryption) is an existing design choice which I don't have influence over (or, might be very difficult to change). Thanks for the suggestion.
    – bdutta74
    Commented Apr 27, 2016 at 12:57

2 Answers 2

Reset to default
2

You can do better than either XML or JSON by exploiting the particular characteristics of your particular data. If the data is completely flat and every row contains the same fields, then CSV is going to be more efficient than either XML or JSON. The reason people generally prefer formats like XML and JSON is because they recognise that the life-time cost of an application and the life-time value of the data is more important than raw machine performance, which is about the cheapest resource in your entire system.

Improve this answer
1
  • +1. I would like to add that, even if his data is flat right now, in further upgrades where data could end up being complex, to use CSV would force to him to recode the stage of key/value mapping. I'm in agreement with @Michael Kay on using CSV but, If It depends on me, I would prefer JSON (lighter than xml) becaus It would be ready for complex data models. It's also the meta-data used in no-Sql DB like Mongo or Redis. To search in JSON data structures is easier than to search in CSV. I like to be cautious
    – Laiv
    Commented Apr 25, 2016 at 9:32
1

The most efficient would probably be a binary format which you could read directly into memory, skipping the parsing step.

Improve this answer
2
  • Most of the time you will convert the data into some kind of object, and that is likely more time consuming than parsing JSON.
    – gnasher729
    Commented Apr 25, 2016 at 21:39
  • @gnasher729 It depends on language. Some provide effective, official or third-party serialisation/deserialisatiion libraries, but no standard format
    – BingLi224
    Commented Sep 25, 2022 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.