Representing a large list of complex objects in JSON

Question

I am currently developing an API that returns JSON structures. One of the domain objects returned is a time series, which is a potentially large list (many thousands of elements) of complex objects.

Say a time series element consists of value and status, there are two ways to represent this in JSON. Either as a list of complex objects representing one element each:

{
    "timeseries": [
        {"value": 1, "status": "OK"},
        {"value": 2, "status": "OK"},
        {"value": 3, "status": "INVALID"},
        // ...
    ]
}

Or as multiple parallel lists:

{
    "timeseries": {
        "values": [1, 2, 3, ...],
        "status": ["OK", "OK", "INVALID", ...]
    }
}

I am wondering if there are any significant advantages of one of those two representations. Right now these seem to be the main differences:

• The list of complex objects seems to be a more natural representation, but is wasteful in that it has to repeat the object keys (value and status) many times, even though they never change.
• The parallel lists seem harder to deserialize back into a list of element objects and slightly error-prone to going out of sync, but represents the data with less repitition.

I am currently leaning towards using a list of full objects under the assumption that sane clients will always consume the API with some form of compression enabled, making the repetition not be an issue in practice. Are there more things I should be aware of to decide for one or the other?

Answer 1

Lean towards option 1, as it's a more expected format.

Option 1 works with JSON as it's designed to be used and therefore benefits from what JSON offers (a degree of human readability, which is good for debugging, and straightforward parsing, which is good for limiting entire categories of bugs to begin with).

Option 2 begrudgingly adopts JSON and subverts many of the benefits. If you don't want human readability, use protobuf or something similar... AIWalker's "CSV"-like approach isn't terrible either. It is marginally better (readable) than splitting objects apart and recombining them. But, this is still not as good (readable) as using JSON "as designed".

Also bear in mind, your API responses are also likely going to be gzipped. Most of the repetition in option 1 will be quickly and transparently condensed over the wire.

As an aside, if you're moving a lot of data, also consider JSONL or paginated results. Pagination can be especially helpful for web clients, as it places natural pauses in the processing, providing a degree of "organic" protection against UI lockups.

Answer 2

A list of objects is easier to work with. You can use append, map, filter... All the nice things JS Arrays have which manual indexing doesn't. And there's no way to get out of sync, so that's an entire class of bugs gone.

If you're worried about efficiency:

• Measure (premature optimization is the root of all evil)
• Consider the list of lists trick AIWalker proposed
• Consider an outright binary format
• Make sure gzip is enabled
• Measure (it's worth saying twice)

Answer 3

Do the easiest version of your api and json first and determine afterwards if it is fast enough. If you’ve got a few hundred samples a straight serialise/deserialise of your Dtos; even if this includes repeated field names, etc. shouldn’t be an issue. If you’ve got long time series; high sampling rates, multiple signals then maybe think about optimising.

The values will always need to be transferred as some sort of array; but your time stamps can be simplified. If it’s a regular series; first timstamp and sampleRate are all you need, but pack any missing values with nulls.

Just remember that any clever deviation from simple serialisation needs to undone in the client :-)

Answer 4

If you are concerned that the amount of data overhead of duplicating the keys is going to be a large enough problem for you that it is worth optimising for, there is another approach you may wish to consider

{
    "timeseries": [
        [1, "OK"],
        [2, "OK"],
        [3, "INVALID"],
        // ...
    ]
}

in this way, you keep objects together but in a structure that does not need to duplicate the keys.

As someone who is just inspecting the data, if it were to happen manually, having an array of values, and an array of statuses, where respective indexes are related (status[1] and values[1] relate), varying value lengths over long durations would make it hard to read for a human. Whether that is a concern to you or not i cant say.

if you want to optimise it even further, you may consider having all the possible statuses be a numeric enum instead, eg:

{
    "timeseries": [
        [1, 0],
        [2, 0],
        [3, 1],
        // ...
    ]
}

where you have a mapping of 0 = ok, 1 = invalid, 2 = some other fail state...etc.

realistically, these optimisations will be of minimal use in most cases due to compression and how large your messages would need to be for these differences to actually amount for much

Answer 5

If the values are unique, you could consider returning an "object"

{
    "timeseries": {
        "1": "OK",
        "2": "OK",
        "3": "INVALID"
        //...
    }
}

I prefer your array version #1, but YMMV.

Answer 6

Having one JSON array for value, and one JSON array for status, might be more efficient. But once you read the data, you will convert it to an array of structs or objects in your programming language. So a JSON array containing dictionaries each with keys "value" and "status" would be easier and implementing it would be easier.

You decide whether a simple implementation is more or less important than the last bit of efficiency. If you have 100 structs, use one JSON array containing dictionaries. If you have 10 million structs, and the operation is time critical, use separate arrays.

Answer 7

Yet another suggestion. Is this really a timeseries of evenly spaced measurements? How about an array? Perhaps with some metadata.

{
   "start": "2021-16-06 07:36:25Z",
   "delta": "ms",
   "values": ["OK", "OK", "INVALID",...]
}

Occasional missing data could be represented by null or "", etc...