Timeline for Is creating and writing to one large file faster than creating and writing to many smaller files in Python on Linux?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 27, 2014 at 8:57 | comment | added | Jan Doggen | @gnuey You could just write a test program and look at the results. Then you have the results for your configuration instead of people having to ask about that configuration. | |
| Aug 27, 2014 at 8:25 | comment | added | Basile Starynkevitch | Unused RAM is lost RAM. You don't want that, and the kernel tries hard to use RAM (at least as file system cache). | |
| Aug 27, 2014 at 8:24 | comment | added | LazerSharks | Ah. In previous runs the system said all the RAM was taken up. So this is probably due to disk caching described at linuxatemyram.com? Do you think dealing with a file larger than RAM will slow down the system, even though it can handle it? | |
| Aug 27, 2014 at 8:19 | comment | added | Basile Starynkevitch | Don't worry in that case. A few gigabytes and a few dozens of thousands of files is not much. But see my answer. | |
| Aug 27, 2014 at 8:17 | comment | added | LazerSharks | In the few gigabytes. Each file would be a few MBs, but there are thousands of files. I think 16gigs of RAM but I'll need to double check on that. | |
| Aug 27, 2014 at 8:06 | comment | added | Basile Starynkevitch | How big is your Linux machine? How much RAM and disk? And how big is your data: gigabytes or petabytes? | |
| Aug 27, 2014 at 8:05 | comment | added | LazerSharks | Indeed. I'm thinking that maybe just having one file open for the main part of the program will speed things up. Although now that I think about it, if as @Phillip Murry says below the OS will flush RAM to hard drive when the single text file gets too big then this might make the program slower? | |
| Aug 27, 2014 at 8:01 | comment | added | mouviciel | In the end, the small files will eventually be created, written to and closed. | |
| Aug 27, 2014 at 7:53 | history | edited | LazerSharks | CC BY-SA 3.0 |
deleted 5 characters in body
|
| Aug 27, 2014 at 7:41 | answer | added | Basile Starynkevitch | timeline score: 7 | |
| Aug 27, 2014 at 7:36 | answer | added | Philipp Murry | timeline score: 1 | |
| Aug 27, 2014 at 7:05 | review | First posts | |||
| Aug 27, 2014 at 8:58 | |||||
| Aug 27, 2014 at 7:02 | history | asked | LazerSharks | CC BY-SA 3.0 |