3

I am currently developing an API that returns JSON structures. One of the domain objects returned is a time series, which is a potentially large list (many thousands of elements) of complex objects.

Say a time series element consists of value and status, there are two ways to represent this in JSON. Either as a list of complex objects representing one element each:

{
    "timeseries": [
        {"value": 1, "status": "OK"},
        {"value": 2, "status": "OK"},
        {"value": 3, "status": "INVALID"},
        // ...
    ]
}

Or as multiple parallel lists:

{
    "timeseries": {
        "values": [1, 2, 3, ...],
        "status": ["OK", "OK", "INVALID", ...]
    }
}

I am wondering if there are any significant advantages of one of those two representations. Right now these seem to be the main differences:

  • The list of complex objects seems to be a more natural representation, but is wasteful in that it has to repeat the object keys (value and status) many times, even though they never change.
  • The parallel lists seem harder to deserialize back into a list of element objects and slightly error-prone to going out of sync, but represents the data with less repitition.

I am currently leaning towards using a list of full objects under the assumption that sane clients will always consume the API with some form of compression enabled, making the repetition not be an issue in practice. Are there more things I should be aware of to decide for one or the other?

6
  • 3
    You're essentially asking for opinions, that's probably the reason someone downvoted this. You already have a very reasonable basis for choosing the first alternative. It is indeed more natural to keep related things together, even at the cost of some memory/bandwidth, as compression takes care of that relatively well. Jun 14, 2021 at 9:57
  • Thank you, I shall go with that then. Are you aware of any networks where open-ended questions like this are allowed?
    – Felk
    Jun 14, 2021 at 10:13
  • 1
    softwareengineering.meta.stackexchange.com/questions/6742/… has some info, but I'd say your question is on the edge as it's not really seeking discussion but "things to consider". Still you might try Quora, Reddit, or some other site or even the whiteboard chat on StackExchange (chat.stackexchange.com/rooms/info/21/the-whiteboard). Jun 14, 2021 at 10:47
  • 1
    Upvoting the question as transfer of time series data is fairly tricky to achieve; especially when users and devs have limited bandwidth at present. Jun 15, 2021 at 23:11
  • 1
    Check out the answer from @user949300 but if that doesn't work consider gzip compression. It works really well when you have the same string repeated many times and is well-supported by browsers.
    – JimmyJames
    Jun 16, 2021 at 21:15

7 Answers 7

2

Lean towards option 1, as it's a more expected format.

Option 1 works with JSON as it's designed to be used and therefore benefits from what JSON offers (a degree of human readability, which is good for debugging, and straightforward parsing, which is good for limiting entire categories of bugs to begin with).

Option 2 begrudgingly adopts JSON and subverts many of the benefits. If you don't want human readability, use protobuf or something similar... AIWalker's "CSV"-like approach isn't terrible either. It is marginally better (readable) than splitting objects apart and recombining them. But, this is still not as good (readable) as using JSON "as designed".

Also bear in mind, your API responses are also likely going to be gzipped. Most of the repetition in option 1 will be quickly and transparently condensed over the wire.

As an aside, if you're moving a lot of data, also consider JSONL or paginated results. Pagination can be especially helpful for web clients, as it places natural pauses in the processing, providing a degree of "organic" protection against UI lockups.

3

A list of objects is easier to work with. You can use append, map, filter... All the nice things JS Arrays have which manual indexing doesn't. And there's no way to get out of sync, so that's an entire class of bugs gone.

If you're worried about efficiency:

  • Measure (premature optimization is the root of all evil)
  • Consider the list of lists trick AIWalker proposed
  • Consider an outright binary format
  • Make sure gzip is enabled
  • Measure (it's worth saying twice)
1

Do the easiest version of your api and json first and determine afterwards if it is fast enough. If you’ve got a few hundred samples a straight serialise/deserialise of your Dtos; even if this includes repeated field names, etc. shouldn’t be an issue. If you’ve got long time series; high sampling rates, multiple signals then maybe think about optimising.

The values will always need to be transferred as some sort of array; but your time stamps can be simplified. If it’s a regular series; first timstamp and sampleRate are all you need, but pack any missing values with nulls.

Just remember that any clever deviation from simple serialisation needs to undone in the client :-)

1

If you are concerned that the amount of data overhead of duplicating the keys is going to be a large enough problem for you that it is worth optimising for, there is another approach you may wish to consider

{
    "timeseries": [
        [1, "OK"],
        [2, "OK"],
        [3, "INVALID"],
        // ...
    ]
}

in this way, you keep objects together but in a structure that does not need to duplicate the keys.

As someone who is just inspecting the data, if it were to happen manually, having an array of values, and an array of statuses, where respective indexes are related (status[1] and values[1] relate), varying value lengths over long durations would make it hard to read for a human. Whether that is a concern to you or not i cant say.

if you want to optimise it even further, you may consider having all the possible statuses be a numeric enum instead, eg:

{
    "timeseries": [
        [1, 0],
        [2, 0],
        [3, 1],
        // ...
    ]
}

where you have a mapping of 0 = ok, 1 = invalid, 2 = some other fail state...etc.

realistically, these optimisations will be of minimal use in most cases due to compression and how large your messages would need to be for these differences to actually amount for much

1
  • I bet both of these structures are faster to serialize and especially deserialize as well as being fewer bytes over the wire (though the OP didn't mention CPU being a concern).
    – davidbak
    Jun 16, 2021 at 23:49
0

If the values are unique, you could consider returning an "object"

{
    "timeseries": {
        "1": "OK",
        "2": "OK",
        "3": "INVALID"
        //...
    }
}

I prefer your array version #1, but YMMV.

0

Having one JSON array for value, and one JSON array for status, might be more efficient. But once you read the data, you will convert it to an array of structs or objects in your programming language. So a JSON array containing dictionaries each with keys "value" and "status" would be easier and implementing it would be easier.

You decide whether a simple implementation is more or less important than the last bit of efficiency. If you have 100 structs, use one JSON array containing dictionaries. If you have 10 million structs, and the operation is time critical, use separate arrays.

0

Yet another suggestion. Is this really a timeseries of evenly spaced measurements? How about an array? Perhaps with some metadata.

{
   "start": "2021-16-06 07:36:25Z",
   "delta": "ms",
   "values": ["OK", "OK", "INVALID",...]
}

Occasional missing data could be represented by null or "", etc...

1
  • What I called value is not a timestamp but an arbitrary value, say a measurement. Therefore this unfortunately does not address the problem
    – Felk
    Jun 18, 2021 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.