4

On our project we have this data format that we use to process and record data on. As of late our application changed so that many of the data formats parameters have become obsolete. Currently we receive these data "packets" over internet (UDP or TCP) and from saved binary files.

We want to create a new more space efficient format, removing things we don't need. Each format is divided into a header and the payload, where things like time-stamp information and some description of the payload is in the header.

To ensure that we can support multiple versions of a format, we decided that it made sense to put some sort of format version ID at the top of the format for every format we make. Unfortunately the previous format (created by people who are no longer on our team) does not follow the convention, and at some point the decision was made to put the format version ID in the middle of the format, in between where all the now useless junk data was.

reading this older format is an issue because currently we actually have gigabytes of that formats data that we use as test data for our application, stuff that was collected in the field.

How do we both ensure formats that don't follow the format format version ID, everything else are still able to be read by our application and future format versions that we create?

We've considered the following:

  • Just moving on to the next format, ignoring old data. Not responsible, prohibitively expensive.

  • Having the user some how specify which format is which (formats which can be found out from header immediately vs old format types). Annoying, and hard on people who are not devs on this project but also contribute (of which there are many).

  • Having new format versions follow old version up to version ID portion. mitigates many of the benefits of moving to the new version, requires careful planning of where to place header bytes to ensure version ID is still in the same location (harder on developers).

  • Converting old formats to version ID first header versions, requires new tooling and maintaining of version converter, requires everyone else's files to be updated as well, these recorded files are with people who are not devs and aren't using version control either, so it will be difficult to make sure already recorded data can be correctly used for everyone.

Here is an example of what the current header looks like:

* = marked for removal

size: 8 bytes
payload metadata: 8 bytes
payload metadata: 8 bytes
* non-standard timeformat: 8 bytes 
* non-standard timeformat: 8 bytes
* legacy undocumented data: 8 bytes
version number: 8 bytes
* source metadata: 8 bytes // may not want this all the time
sequence number: 8 bytes
short range time: 8 bytes
payload metadata: 8 bytes
* size data?: 8 bytes
* spare data: 8 bytes
payload: N bytes
  • You may want to look at some examples of how existing binary formats solve this problem: martin.kleppmann.com/2012/12/05/… – Joeri Sebrechts Jan 11 '18 at 15:02
  • 1
    How is your current format structured? Is it possible to generate a malformed message under the current format, which could then be used to identify the new versions? Or will any sequence of bytes be accepted? Where is the version currently stored? At a fixed offset? Your options 2 and 3 seem most desirable, if possible. – amon Jan 11 '18 at 15:08
  • @amon I've given an example of the current format. – opa Jan 11 '18 at 15:20
  • Is the "binary" requirement an inevitable decision? Note that you were already bitten once by someone's decision against a better (in the sense of business value) solution - you are paying now 10x the upfront costs which they thought to have saved. If you want an interesting sidestep from the usual suspects in data exchange formats, look at the Universal Binary Format which was invented by one of the creators of Erlang and is a good compromise between a binary protocol and a pure textual one IMHO. – Vroomfondel Jan 11 '18 at 16:19
  • Is the header on all existing formats always the same size? You mention that at some point the version number was added in the middle, was the header smaller before then? If so, how do you know the size of the header? Or is it always the same size and something else was in that location previously? Also how big is the header compared to the typical payload? – Mr.Mindor Jan 11 '18 at 19:29
11

It seems to me that the simplest solution is to make your version header unambiguous and make sure that the old format can never look like it has a format header, you simply look for it. If it's not there you assume it's the old style and try to find it from the middle. There might also be things in the beginning of the old format that can clue you in.

The key here is that you need to find some sort of scheme for your version preamble that the old format cannot produce. For example, let's assume the old version never starts with a 0 byte. You could start your preamble with 0x00 0x00 0x00 0x00. Then when you start reading the data, you read in the first 4 bytes and if there's any non-zero value, you are looking at an old version (or a bad request.) An example of this being done is in UTF-8 and its backwards compatibility with ascii.

  • Use of some kinda of GUID might help here but it doesn't have to be fancy. 0xCAFEBABE was good enough for Java. – candied_orange Jan 11 '18 at 15:27
  • I do not believe this is possible, like at all, given the old format (which I've updated to include). I don't see anything that can "clue me in" unambiguously, especially seeing as so much stuff changes. – opa Jan 11 '18 at 15:29
  • @snb Really? Why not? – candied_orange Jan 11 '18 at 15:30
  • 1
    For example, Let's say the old version never starts with a 0 byte. In the newer versions, you start the version header with 4 '0' bytes. Then your old version will never look like it has a header. You just need to find a prefix that the old version can never produce. Ralf gives a potential scheme. – JimmyJames Jan 11 '18 at 16:04
  • 2
    Ehm, starting with a BOM is making UTF-8 intentionally backwards-incompatible to ASCII. – Deduplicator Jan 11 '18 at 21:20
9

Your old format starts with an 8-byte "size" element (probably a 64-bit long). If that's the file size in bytes (what I guess), it most probably never exceeded 100 GByte, meaning 2^37. So, if you offset your new version numbers by e.g. 2^40 and store them as the first 8 bytes, it'll be easy to discriminate:

  • Read the first 64-bit word as "version".
  • Version below 2^40: it's an old file, branch to the old reader that will re-read or re-interpret the initial bytes as size.
  • Version above 2^40: branch to the new reader appropriate for this version.
  • 3
    Or if the size includes the header, then it can never correctly be all-zero. For that matter, an all-zero size field might be known-bad for the old format even if it did not include the header size. – John Bollinger Jan 11 '18 at 20:53
6

The convention format version ID, everything else is good because it gives maximum flexibility for the format. In your case the version ID isn't right at the front but at a fixed offset. This slightly reduces your flexibility but not fatally so: as long as your new format keeps the version at the same offset, you can keep going.

Your header parsing code will need to check the version number first, and then dispatch to the new or old parsing implementation:

Header parse_header(char const* buffer) {
  int version = *(int const*)(buffer + VERSION_OFFSET);
  if (version < NEW_VERSION)
    return parse_header_legacy(buffer);
  else
    return parse_header_new(buffer);
}

You can then reshuffle the header fields to use the space as you see fit. This likely does prevent a self-describing data format as you still need a fixed-size layout. But this is not necessarily a huge problem. If necessary, you can implement a header extension mechanism that stores self-describing header fields in the payload section after the header.

The social costs of such a compatible header upgrade are minimal, given the alternatives. Your new code will be able to receive both new and old packets, though old code will not be able to read new packets. If you encapsulate the parsing code in routines that represent the header with a common data format, this format change is not a burden on contributors: they use the parsed data structure, not the raw packet bytes. The only reason why this might not be acceptable if you truly need zero-overhead parsing, i.e. reinterpreting the raw bytes as a struct.

-2

You should really take a look at Protocol Buffers. Chances is you'll like it and it solves your issues.

The idea in the newest data format version is that "required" fields are no longer supported, since you'll anyway someday change a format in such a manner that a field that used to be required is no longer required.

As long as you don't go changing types or identifiers of existing fields, the format is fully backwards and forwards compatible. Nevertheless, it's a genuine binary format and encoding and decoding is quite fast.

Although the name says "protocol", you can use them well even for storing information to disk. So there's nothing protocol specific in this.

There are bindings for very many languages. If you use a language that has any kind of reasonable user base, you'll probably find protocol buffers support for your language.

Of course, if your legacy format wasn't based on protocol buffers, you may need to have support for reading legacy data in addition to protocol buffers data during the transition period.

  • 4
    While using such a format would prevent some kinds of compatibility problems in the future, this doesn't solve backwards compatibility with the existing format: given a buffer of bytes, is that a legacy packet or a protobuf message? There's no way to tell, because the legacy message may appear to look like a valid protobuf encoding. – amon Jan 11 '18 at 20:22
  • Not to mention protocol buffers are not supported in the language we are using.... – opa Jan 11 '18 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.