What is the best way to save data of a C++ Program ? Binary vs JSON serialization

Question

We are implementing a CAD program (in C++, Qt) where we have interdependant classes : The smallest brick is the Pattern, it is just a distribution of points. Then we have Layouts which contain Patterns at a given height (z = constant). And finally Stacks, which are a stacking of Layout.

All these entities have a unique id and can also have a list of masks which are applied to it.

Knowing that each of the classes are linked (std::vector of pointers everywhere) by a tree structure, what would be the best way to save a project countaining several stacks ? The objective of saving would be to later load again the project and modify it.

We already looked into Boost and Qt Serialization but it seems to me that binary Serialization may be way too much for what we want to do (as it needs time to be integrated). On the other hand we tought about an XML/JSON serialization, having something which would look like this :

<Layout>
 <id> 12</id>
 <z> 0</z>
 <patterns>
  <pattern id> pattern0</pattern id>
  <pattern id> pattern1</pattern id>
 </patterns>
</Layout>

Are they some other ways ? What would be the best one and why ? We lack of objectivity on this point.

Answer 1

First thing I would check is if using an existing CAD format would not be sufficient. That would not only prevent you from reinventing the wheel, but also improve the interoperability with other CAD programs. Moreover, adapting the terms of such a format like "Layers" and "Entities", instead of using terms like "layouts" and "patterns" (which have typically different meanings in other CAD systems), you might be able to reduce communication issues.

If you really feel you need to create your own file format, make sure you can keep the format backwards compatible with each new version of your CAD system, otherwise you will run sooner or later into problems. This is much simpler when your format provides a lot of metadata. In my experience, formats like XML or JSON make it easier to fulfill this requirement than proprietary binary formats. The decision between these two is surely pretty opinonated. XML is still more wideley used and supported by a lot more third party tools. It will allow you easier integration of other existing XML based standards. On the other hand, it has the drawback that the files tend to become very big, much bigger than a typical JSON file carrying the same information. This is a tradeoff you need to consider for your use case.

Answer 2

I've had great success using SQLite databases as a file format for graphics applications. Its both very reliable, and being a standard format, its contents are easily viewed/converted by external programs.

Answer 3

JSON is simple and portable. External tools won't have any problems reading or writing it. You won't have any problems yourself reading and writing the data on any architecture, which is very nice. It is also very easy to be compatible with older and newer version, often just by ignoring items that you don't understand.

JSON is a bit verbose if you store a lot of dictionaries with long keys. For example if you store 1000 longitude/latitude pairs you could store it like

[
  { "longitude": 10.29340, "latitude": 35.34989 },
  { "longitude": 10.29340, "latitude": 35.34989 },
  ... repeat 998 more times
]

If you have so much data that this is negatively affecting you, you can as well store the data more compact as

{ "latitudes": [10.29340, 10.29340, .. 998 more values], 
  "longitudes": [35.34989, 35.34989, ... 998 more values]
}

Another nice thing is that you can store data in a more compact or more readable format. (Compact by removing line separators, spaces).

JSON can store arrays, dictionaries (key-value pairs), any Unicode strings, numbers, booleans and null values. It cannot store dates but there is a standard for storing dates as strings in JSON.

If you are worried about version compatibility, I'd recommend adding one key/value pair specifiying the number of the spec used for writing the data, and one key/value pair specifying the number of the lowest spec that can process the data by ignoring information that it doesn't know about.

Answer 4

Ignoring the specific CAD issue, but taking into account that this use case could potentially generate rather large files, I would recommend looking into protobuf. It's a rather compact binary format with easy versioning for your future needs:

... language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Disclaimer: We use it for message serialization, not for file storage.

Caveat: One limitation I know for protobuf is that a single message max approx. size is 64MB. So a file format such as yours should normally be designed as a stream of multiple protobuf messages instead of one "huge" message per file.

Answer 5

XML and to a lesser extent JSON is probably the best options out there if you want a human readable format.

However, human readable also means human editable, which in turn means having people make changes that are invalid and then having to provide support to allow these humans to figure out their mistakes.

Both XML and JSON do not solve the format extensions that will occur over time, allowing newer programs to work on older versions and vice versa.

I would also consider Google protocol buffers as a good choice for a binary format option. It has good support in C, generates smaller data files than JSON and will allow the reading of different data format versions.

It will allow your team to focus on the data structure, and to manage format changes comfortably.