i need to store a large amount of data - about 10 million entries of the format

    unique hash (64 chars), value 1 (5 chars), value 2 (9 chars)

i will be reading and deleting (but not updating) this data in quick succession. for example, i might write 10 lines, then read 3 and delete 3 lines, before moving on to more writes and reads...

its a single threaded desktop program so reads are only initiated after writes and deletes have been completed. i would like to avoid using a database so that the data can be human-readable from the file system and so that people who download and use my program can see it and inspect it for themselves. i expect my users to be running a linux os and my program is written in python.

i have come up with three possible options for how to store this data:

**option 1**

put the data in a single file. since the hashes are unique i can arrange them in ascending order like an index so as to give `log2 N` read times. deleting and inserting will happen often and would slow the process down as `N` gets large.

**option 2**

use the unique hash as a file name and put the values inside this file. this would be my preferred option so long as there are no obvious performance drawbacks? i know some file systems [slow down when a large number of files exist in a single directory][1]. and i really can't make any assumptions about which file system the users of my program will be using. i need to choose a solution which is quick on all file systems.

**option 3**

use the unique hash to create a directory to store the values in. so for example, if a record has hash `abcd` then i would create dir and file `base_dir/a/b/c/d.txt`. does this get around the file system performance issue issue? or is this just as bad as option 2 depending on the type of file system the user is running?

**question**

so really i'm just looking for help to decide which option to pick. also if you have any other options that are superior to the ones i have listed here i would like to hear them. maybe some database software like mysqli that sores data in human-readable text? that would be ideal really, but i imagine that most databases would use binary for efficiency...

  [1]: https://serverfault.com/questions/43133