0

I need to write continuously into a JSON file in C++. The format and structure are defined as requirement. The data is produced in the process itself and needs to be stored multiple times per second. It's actually a protocol of realtime data.

Structure is basically: [{Object01}, {Object02}, ...]

With every tenth of a sec, I have to add another dataset object. The total duration is between 30 to 90 minutes.

I'll go for rapidJSON, to ensure the proper form, etc.

The question is: How do I organize the write stream?

  1. I hold all the data in one Document object and write it into the file at the end of the process.
  2. I need to read in the whole file in every loop and write it back.
  3. There is a proper way that only new objects getting appended to the array in the file.

The I would go for the last point, but I have actually no idea how to do that.

3
  • 2
    Why JSON (which is inherently unsuited and inefficient for large amounts of data)? Do you need this to be a single file of would you be better of with one file per object that can be easily addressed by file name later? Sep 23 at 6:13
  • @Martin Maat: It's a given requirement. I'm totally aware of and you're right. Also the single file structure is given.
    – da-chiller
    Sep 29 at 5:27
  • I see that you tried to answer a few of my former questions. Still I think it is unclear if #1 is an option for your case or not, it would probably be the most simple way to implement the requirements you described. The drawback would be the required memory (for ~50K-60K objects), and the fact the protocol will not be externally available before the process ends (and if the process crashes in between, there will be no protocol written at all). If those drawbacks are not an issue for your case, go with #1 (else use Bart's solution).
    – Doc Brown
    Sep 29 at 5:55
6

The last time I had a requirement like that, the process looked like

  1. When starting a new file, write the characters [\n to it to open a JSON array.
  2. When appending to an existing file, seek to the end. Then move 3 characters back (over the \n]\n sequence expected at the end) and write ,\n
  3. Serialize a single object to JSON
  4. Write ]\n or \n]\n to the file (depending on if the serializer ends the string with a newline or not)

This means that your JSON library only processes a single object at a time and you are manually maintaining the file as a valid JSON array.

1
  • That'll probably the way to go. Thanks
    – da-chiller
    Sep 29 at 5:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.