If using Python on a Linux machine, which of the following would be faster? Why?

  1. Creating a file at the very beginning of the program, writing very large amounts of data (text), closing it, then splitting the large file up into many smaller files at the very end of the program.
  2. Throughout the program's span, many smaller files will be created, written to and closed.

Specifically, the program in question is one which needs to record the state of a very large array at each of many time-steps. The state of the array at each time-step needs to be recorded in independent files.

I've worked with C on Linux and know that opening/creating and closing files is quite time-expensive, and fewer open/create operations means faster programs. Is the same true if writing in Python? Would changing the language even matter if still using the same OS?

I'm also interested in RAM's role in this context. For example -- correct me if I'm wrong -- I'm assuming that parts of a file being written to will be placed in RAM. If the file gets too big, will it bloat RAM and cause problems in speed or other areas? If an answer could incorporate RAM that would be great.

  • 1
    In the end, the small files will eventually be created, written to and closed. – mouviciel Aug 27 '14 at 8:01
  • Indeed. I'm thinking that maybe just having one file open for the main part of the program will speed things up. Although now that I think about it, if as @Phillip Murry says below the OS will flush RAM to hard drive when the single text file gets too big then this might make the program slower? – LazerSharks Aug 27 '14 at 8:05
  • 1
    Don't worry in that case. A few gigabytes and a few dozens of thousands of files is not much. But see my answer. – Basile Starynkevitch Aug 27 '14 at 8:19
  • 3
    Unused RAM is lost RAM. You don't want that, and the kernel tries hard to use RAM (at least as file system cache). – Basile Starynkevitch Aug 27 '14 at 8:25
  • 1
    @gnuey You could just write a test program and look at the results. Then you have the results for your configuration instead of people having to ask about that configuration. – Jan Doggen Aug 27 '14 at 8:57

To answer your question, you really should benchmark (i.e. measure the execution time of several variants of your program). I guess it might depend on how many small files you need (10 thousand files is not the same as 10 billion files), and what file system you are using. You could use tmpfs file systems. It also obviously depends on the hardware (SSD disks are faster).

I would also suggest to avoid putting a big lot of files in the same directory. So prefer dir01/file001.txt ... dir01/file999.txt dir02/file001.txt ... to file00001.txt ... file99999.txt ie have directories with e.g. at most a thousand files.

I would also advise to avoid having a big lot of tiny files (e.g. files with less than a hundred bytes of data each): they make a lot of filesystems unhappy (since a file needs at least its inode).

However, you should perhaps consider other alternatives, like using a database (which might be as simple as Sqlite ...) or using some indexed file (like gdbm ...)

Regarding RAM, the kernel tries quite hard to keep file data in RAM. See e.g. linuxatemyram.com; read about posix_fadvise(2), fsync(2), readahead(2), ...

BTW, Python code will ultimately call C code and use the same (kernel provided) syscalls(2). Most file system related processing happens inside the Linux kernel. So it won't be faster (unless Python adds it own user-space buffering to e.g. read(2) data in megabyte chunks, hence lowering the number of executed syscalls).

Notice that every Linux system is able to deal with a lot of disk data, either with a single huge file (much bigger than available RAM: you could have a 50Gbyte file on your laptop, and a terabyte file on your desktop!) or in many files.

  • I'm guessing it'll much larger than 10 thousand, I'll check on that. It's a pretty big simulation, which has already taken a few weeks with some previous runs. So do you think the greater amount of small files means greater time? Didn't think of using a DB. Thanks for the suggestions! – LazerSharks Aug 27 '14 at 7:48
  • Also, what do you mean by benchmark? Just timing the program using time functions? – LazerSharks Aug 27 '14 at 7:55

I think it doesn't so much depend on the programming language, but on how Linux (and other systems) handle files: For each file that's created, an inode is created that contains meta information about the file. Therefore it is faster creating one large file than a myriad of smaller ones.

Regarding RAM, the OS should take care of it anyway. If too many pages get occupied, the OS writes them to the hard drive. If you want to handle it on your own, there's a flush function in python as well: http://www.tutorialspoint.com/python/file_flush.htm

  • Ah, so RAM is indeed being filled up when writing to single file. Awesome, thank you for pointing to the Python flush function! – LazerSharks Aug 27 '14 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.