If using Python on a Linux machine, which of the following would be faster? Why?
- Creating a file at the very beginning of the program, writing very large amounts of data (text), closing it, then splitting the large file up into many smaller files at the very end of the program.
- Throughout the program's span, many smaller files will be created, written to and closed.
Specifically, the program in question is one which needs to record the state of a very large array at each of many time-steps. The state of the array at each time-step needs to be recorded in independent files.
I've worked with C on Linux and know that opening/creating and closing files is quite time-expensive, and fewer open/create operations means faster programs. Is the same true if writing in Python? Would changing the language even matter if still using the same OS?
I'm also interested in RAM's role in this context. For example -- correct me if I'm wrong -- I'm assuming that parts of a file being written to will be placed in RAM. If the file gets too big, will it bloat RAM and cause problems in speed or other areas? If an answer could incorporate RAM that would be great.