I wrote an application that tests the performance of evolutionary algorithms. This application performs a run of the algorithm which consists of several generations. The data which is produced by my application looks like this:

run1.run             // text file containing metadata
run1_data            // folder containing experimental data
   -statistics1      // text file containing some specific statistics
   -generation10     // snapshot of the algorithm at generation10

Once this data is written, it is never changed.

When I want to examine the data, my application reads the metadata file (.run), opens the _data directory and reads the rest of the data.

This was all fine until recently. I now have hundreds of thousands of these files and I ran out of inodes on my system and also the loading of the data and copying is extremely slow, even though there are only a few gigabytes. My data seems to be too fragmented, since the files are quite small.

My application is written in C++ and uses the Qt library for filesystem operations. I was thiking of using the <system> header to issue a tar command to archive data after writing and un-archive before reading, but I found out that tar must read the entire archive to find the contents. This is a problem for me, since to save operating memory and time I sometimes load only Statistics1, sometimes only generation10...

I was considering to change the format of my data so that there would be only one file, which would have something like a table of contents at the beginning, followed by all data files concatenated. The table of contents would indicate the beginning and end of each concatenated file. However I am not sure if this is a good solutions since the std::ifstream class that I use to read the files cannot make random jumps.

I am a beginner programmer and I do not want to waste a lot of time on developing something that would not work so I ask for any advice on how to solve my problem.

  • 2
    Have you tried using libzip? I'm fairly sure zip files support random access, and you're not forced to compress the files. – Doval Jun 24 '14 at 16:32

You could consider using some indexed file library like gdbm (or something else).

You could also perhaps consider using sqlite (it is a bit of overkill, but learning some tiny SQL skills is useful!) - or even using a real database system (e.g. postgresql or mongodb). Don't forget to backup & dump the data in database (i.e. SQL) format.

You might also be interested in textual serialization formats like JSON (there are libraries for them, e.g. jsoncpp, and it is nice to handle textual data). You could put JSON data inside GDBM or sqlite containers (see this example of mine).

BTW, if you want to keep having many files, perhaps organizing them in directories (i.e. dir01/data0020 ....) might help.

You probably would want to make a helper application to browse or access your data... Think also about backing up your data in textual (not binary) format!

There are some libraries handling tar format like libtar but I guess you should not use them.

Look also if your field (evolutionary algorithms) has not defined some conventions or formats. Document your format (even for yourself!).

  • Thank you! I will need to study all the links you provided. I already use JSON for some of my metadata, but I think JSON also needs to read the entire file to construct the JSON object. – Martin Drozdik Jun 24 '14 at 16:54
  • 1
    I'll go for the sqlite solution. – Martin Drozdik Jun 24 '14 at 17:02
  • 2
    Then, for performance and other reasons, consider grouping several SQL requests in a single transaction (even for sqlite). – Basile Starynkevitch Jun 24 '14 at 17:16
  • 1
    I vote for SQLite. And since you are already using Qt, you can resort to Qt's SQL module instead of the native, plain C API of SQLite. – Siyuan Ren Jun 25 '14 at 8:01

You can also use a zip file. It's like tar, but the index is held at the end of the file, and is found by seeking to the end of the file.

If you don't compress when creating the zip, then it's just like a tar, but with the abilily to seek to any file inside, and you can even mmap the entries.

It's better than sqlite because it's easy to browse a zip file, and many file utilities can treat a zipfile like a folder. BeyondCompare can even compare two zip files just like comparing two folders.


The sqlite suggestion is good if you believe that your data is relational. You can squeeze the data into sqlite if even if it is not relational, however, a nosql database may be a better choice. For example Mongodb might be a good choice or redis may be good choices.

By the way, fstream is fine with random access. See seekg.

  • sqlite can also be used without truely relational data. – Basile Starynkevitch Jun 25 '14 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.