I wrote an application that tests the performance of evolutionary algorithms. This application performs a run of the algorithm which consists of several generations. The data which is produced by my application looks like this:
run1.run // text file containing metadata run1_data // folder containing experimental data -statistics1 // text file containing some specific statistics -statistics2 -generation0 -generation10 // snapshot of the algorithm at generation10 -generation20 ...
Once this data is written, it is never changed.
When I want to examine the data, my application reads the metadata file (.run), opens the _data directory and reads the rest of the data.
This was all fine until recently. I now have hundreds of thousands of these files and I ran out of inodes on my system and also the loading of the data and copying is extremely slow, even though there are only a few gigabytes. My data seems to be too fragmented, since the files are quite small.
My application is written in C++ and uses the Qt library for filesystem operations. I was thiking of using the
<system> header to issue a
tar command to archive data after writing and un-archive before reading, but I found out that
tar must read the entire archive to find the contents. This is a problem for me, since to save operating memory and time I sometimes load only
Statistics1, sometimes only
I was considering to change the format of my data so that there would be only one file, which would have something like a table of contents at the beginning, followed by all data files concatenated. The table of contents would indicate the beginning and end of each concatenated file. However I am not sure if this is a good solutions since the
std::ifstream class that I use to read the files cannot make random jumps.
I am a beginner programmer and I do not want to waste a lot of time on developing something that would not work so I ask for any advice on how to solve my problem.