3

We are implementing a CAD program (in C++, Qt) where we have interdependant classes : The smallest brick is the Pattern, it is just a distribution of points. Then we have Layouts which contain Patterns at a given height (z = constant). And finally Stacks, which are a stacking of Layout.

All these entities have a unique id and can also have a list of masks which are applied to it.

Knowing that each of the classes are linked (std::vector of pointers everywhere) by a tree structure, what would be the best way to save a project countaining several stacks ? The objective of saving would be to later load again the project and modify it.

We already looked into Boost and Qt Serialization but it seems to me that binary Serialization may be way too much for what we want to do (as it needs time to be integrated). On the other hand we tought about an XML/JSON serialization, having something which would look like this :

<Layout>
 <id> 12</id>
 <z> 0</z>
 <patterns>
  <pattern id> pattern0</pattern id>
  <pattern id> pattern1</pattern id>
 </patterns>
</Layout>

Are they some other ways ? What would be the best one and why ? We lack of objectivity on this point.

Improve this question
3
  • How large are those objects? XML/JSON adds quite some additional size, sometimes more than the data itself. Otherwise you already listed the advantages, it's human readable, there are lots of libraries available, easy to implement etc. – thorsten müller May 2 '16 at 10:19
  • I wad deleting my post when you commented... For Pattern for exemple I have id (int), name (string), pointer to type of Point Distribution, list of points (which store because it can be computed from the point distribution), list of (pointers of) masks. – ElevenJune May 2 '16 at 10:22
  • 2
    Whoever voted to close as too broad: I think this question is reasonably answerable. A very clearly defined use case is given and not just a 'what's better?' question. – thorsten müller May 2 '16 at 10:22
6

First thing I would check is if using an existing CAD format would not be sufficient. That would not only prevent you from reinventing the wheel, but also improve the interoperability with other CAD programs. Moreover, adapting the terms of such a format like "Layers" and "Entities", instead of using terms like "layouts" and "patterns" (which have typically different meanings in other CAD systems), you might be able to reduce communication issues.

If you really feel you need to create your own file format, make sure you can keep the format backwards compatible with each new version of your CAD system, otherwise you will run sooner or later into problems. This is much simpler when your format provides a lot of metadata. In my experience, formats like XML or JSON make it easier to fulfill this requirement than proprietary binary formats. The decision between these two is surely pretty opinonated. XML is still more wideley used and supported by a lot more third party tools. It will allow you easier integration of other existing XML based standards. On the other hand, it has the drawback that the files tend to become very big, much bigger than a typical JSON file carrying the same information. This is a tradeoff you need to consider for your use case.

Improve this answer
3
  • Thanks for you answer. We can't use an existing CAD format, we are trying a new approch and we don't want to limit ourselves :) Also thanks for you remark on compatibility, we already thought about it but as we still have time before the software goes to some clients we do not have to manage this issue right now. Right now I'm trying JSON, let's see how it goes ! – ElevenJune May 2 '16 at 15:47
  • 2
    Regarding the size concerns Doc Brown brought up: text formats tend to be larger than binary formats due to encoding of e.g. integers using more bits. If you go the text route, I suggest compressing your output. Even something as simple as running your save file through zlib on the way to disk can help. – user22815 May 2 '16 at 16:39
  • Thanks, we thought of that too but didn't put it in place yet. We wanted to see with more practice if it was really necessary, but if it is that simple why not do it now. – ElevenJune May 3 '16 at 8:39
8

I've had great success using SQLite databases as a file format for graphics applications. Its both very reliable, and being a standard format, its contents are easily viewed/converted by external programs.

Improve this answer
5
  • Do you have more info please ? I don't see what the file could looks like when it is stored. – ElevenJune May 3 '16 at 8:33
  • The file itself is an sqlite file. You don't have to worry about what the file structure looks like because the sqlite engine handles managing it. You read from & write to the file using SQL. – GrandmasterB May 3 '16 at 15:58
  • @ElevenJune one advantage of this approach is that you can read portions of the data as needed, and update existing data without overwriting the entire file. So you could append revisions to the file, or read specific revisions without loading the entire file. – GrandmasterB May 3 '16 at 16:09
  • By what is in the file ? Some binary with all the pointers ? – ElevenJune May 5 '16 at 0:04
  • @ElevenJune its a binary file in SQLite's own database file format. See sqlite.org – GrandmasterB May 5 '16 at 2:37
5

JSON is simple and portable. External tools won't have any problems reading or writing it. You won't have any problems yourself reading and writing the data on any architecture, which is very nice. It is also very easy to be compatible with older and newer version, often just by ignoring items that you don't understand.

JSON is a bit verbose if you store a lot of dictionaries with long keys. For example if you store 1000 longitude/latitude pairs you could store it like

[
  { "longitude": 10.29340, "latitude": 35.34989 },
  { "longitude": 10.29340, "latitude": 35.34989 },
  ... repeat 998 more times
]

If you have so much data that this is negatively affecting you, you can as well store the data more compact as 

{ "latitudes": [10.29340, 10.29340, .. 998 more values], 
  "longitudes": [35.34989, 35.34989, ... 998 more values]
}

Another nice thing is that you can store data in a more compact or more readable format. (Compact by removing line separators, spaces).

JSON can store arrays, dictionaries (key-value pairs), any Unicode strings, numbers, booleans and null values. It cannot store dates but there is a standard for storing dates as strings in JSON.

If you are worried about version compatibility, I'd recommend adding one key/value pair specifiying the number of the spec used for writing the data, and one key/value pair specifying the number of the lowest spec that can process the data by ignoring information that it doesn't know about.

Improve this answer
1
  • Thanks for your answer ! I'm testing JSON right now with your second option more compact. It works well. – ElevenJune May 2 '16 at 15:49
2

Ignoring the specific CAD issue, but taking into account that this use case could potentially generate rather large files, I would recommend looking into protobuf. It's a rather compact binary format with easy versioning for your future needs:

... language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Disclaimer: We use it for message serialization, not for file storage.

Caveat: One limitation I know for protobuf is that a single message max approx. size is 64MB. So a file format such as yours should normally be designed as a stream of multiple protobuf messages instead of one "huge" message per file.

Improve this answer
1
  • Thanks ! I never heard of that, I'll look into it to see if it canfit our needs ! – ElevenJune May 3 '16 at 8:37
1

XML and to a lesser extent JSON is probably the best options out there if you want a human readable format.

However, human readable also means human editable, which in turn means having people make changes that are invalid and then having to provide support to allow these humans to figure out their mistakes.

Both XML and JSON do not solve the format extensions that will occur over time, allowing newer programs to work on older versions and vice versa.

I would also consider Google protocol buffers as a good choice for a binary format option. It has good support in C, generates smaller data files than JSON and will allow the reading of different data format versions.

It will allow your team to focus on the data structure, and to manage format changes comfortably.

Improve this answer
7
  • Thanks for your answer. We thought about the human editable problem. For now we thought about encryption, compression or checksum. We will see later if we want to block the files from being edited (for example if we suppose the users know what they do, my previous compagny was thinking that way, and it is not that stupid when you work with professional people) – ElevenJune May 3 '16 at 8:37
  • I'd say that JSON is much more human readable than XML. – gnasher729 May 3 '16 at 9:00
  • For a non trivial data file, having an XML scheme to validate the data structure has some big advantages over JSON's more relaxed attitude. There is no perfect answer, just a decision on which issues are more important to the project – Michael Shaw May 3 '16 at 9:06
  • JSON of course also works with schemas... – Jon Chesterfield May 3 '16 at 9:08
  • I personally would say that using Protocol Buffers to handle the future versioning issues trumps the argument for a human readable format when the OP is considering encrypting the file anyway. – Michael Shaw May 3 '16 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.