My data:

  • Size - 15+ Terabytes of text.
  • Data: Maximum of 1300 rows (always same row headers) , Maximum of 6 columns
  • The column headers will be different but the value are all percent values that will be between 0 - 100. The total of the columns in one row will add up to 100% always. So we can get rid of one column right away.

Can I keep this as a flat file system? This will be a tree structure, so I don’t think there is any value in MySQL. Maybe NoSQL. Once the data is there, it never has to be updated. It’s just read. Example:

Item, action1, action2, action3
A, 20, 40, 60
B, 20, 30, 50
ZZZ 50, 50, 0


The data will be read on a site. 500 users maximum (most likely under 100) and they will all be reading different files, rarely the same one at the same time. When reading a file all the info is used, so there is no value in being able to only read one row. We will always load all info from the file. The info is only read, never updated.


  1. Best way to store this info?
  2. Each node (aka csv file) in the tree is in its own folder, with sub folders representing the children node in it too. Going all the way down the tree. Is this a massive waste of space? Not sure of a better way to do it if it is.

I'm going to be creating a 15+ TB of data. Very easily will grow to 50tb. I really want to make sure I do this correctly from the beginning. Any advice at all from someone experienced in this can save me a massive amount of time and future headaches. Even hints that will point me in the correct direction I was going to use something like MongoDB but I am starting to think that maybe keeping it as a flat file system won't be bad since I always load the full file anyway and the folder structure keeps track of the tree. No one will need to jump to some node really far down the line.

This is a massive game-tree (more like this). So we start at the root. The root will be a folder called root and have a csv file like above. This folder will have sub folders that represent the children also with csv files like above and folder representing their children. And so on... So to get info you click down the tree. There will be code that just reads the sub folders and creates links to the next nodes. (One game tree will have 25-100k nodes... so that means 25-100k folders which i think may be a waste of space.)

  • 1
    What is the access pattern to select the csv file of interest? Is the file path given? Is some kind of search through the contents necessary? Aug 24, 2017 at 20:50
  • What are you trying to achieve after storing this data? are you will query those values and require fast retrieval time? or it is for archiving purposes? is this a state/action value matrix that needs to be updated regularly?
    – A.Rashad
    Aug 24, 2017 at 20:58
  • 2
    You do this "correctly from the beginning" when you use the most simple solution you can think of which has no obvious drawbacks (seems to be flat files in a folder hierarchy), and try out how it works. There is still room for optimization in performance (for example by picking the right kind of file server storage system) or disk space (for example by compressing each file).
    – Doc Brown
    Aug 24, 2017 at 21:00
  • 2
    ... If it turns out you need something more sophisticated, you did not really invest too much effort. This will be much better than the other way round, by starting with something more complicated "just in case" first, which then might turn out to be a dead end.
    – Doc Brown
    Aug 24, 2017 at 21:04
  • 2
    @lessharm: don't worry about making something working more efficient first, make it working first. Then, measure if you need more efficiency.
    – Doc Brown
    Aug 26, 2017 at 7:11

1 Answer 1


File systems are very well optimised, so flat files might work very well. Since it is a game tree, I imagine each user is walking the tree, choosing a child node each time.

You haven't mentioned the request rate. If it is high, you will want an SSD, which is great for write once read many.

Given that a user always loads the entire file, it would be good to ensure that the file fits in a single block in the file system. The simplest way is to make a custom volume if necessary. This might also push you to use a binary encoding of the file rather than ASCII. Given that the row labels are constant you don't need to store them at all.

Are the values in each file arbitrary? Or can then be calculated based on the values in the parent? If so, it might be better to recalculate on demand - the instruction path length for I/O is quite long so you can do quite a lot of computation.

  • That's exactly how the user will navigate, it will always be one step up or down. The request rate wont be very high and it will be terabytes of data so I wont go with an ssd. Recalculation on the fly is a good idea but not an option at the moment... maybe as my programming skills improve :D What would the value be of the custom volume? I guess most csv files will be around 2kb. I quickly tried a google search on custom volumes and didn't really find much info on it. Maybe I'm using the wrong key words?
    – lessharm
    Aug 28, 2017 at 22:45
  • If you use a new "advance format" disk, the standard sector size is 4K, so each file should fit in a single sector. If you have an older disk reformatting to that size would be advisable. Sep 4, 2017 at 16:05
  • thx for the the advice, looks like win server 2012 uses 4k disk size
    – lessharm
    Sep 6, 2017 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.