1

I hope this is not off topic.

I have to save output from a command to file, but only if length of this output is positive. I've thought about two solutions:

  • save output to a python variable, check length, if positive save to destination file;
  • save output to a temporary file, check length, if positive rename to destination file.

Is there any best practice for this little problem, or which one usually performs better between my proposed solutions?

Typical configuration:

  • file length is usually 1KB
  • we are not using ssds, usually 7200rpm hard disk
  • average RAM is usually 10-20 GB
  • I don't know if filesystem is important, anyway ext3

I know this is a bit unclear, but I was hoping there was some best practice to apply in this case.

Thanks

5
  • 1
    Try them both, profile it, select the best one. The answer likely depends highly on your environment (machine (how much memory? how fast a drive?), input (few hundred bytes? few kilobytes? megabytes? giga?), code (sucky vs my name is Guido)).
    – user40980
    Commented May 1, 2013 at 21:00
  • Thanks.. I'll try to edit my question to reflect this parameters.. My aim was only to know if there were some best practices..
    – gc5
    Commented May 1, 2013 at 21:07
  • ...but only if length of this output is positive.. Does that mean it's possible to have output whose length is negative? Commented May 1, 2013 at 21:16
  • :D the other one possibility is zero of course
    – gc5
    Commented May 1, 2013 at 21:17
  • 1
    A 1kB string isn't that large when it comes to languages such as python (or perl) where processing files is a common occurrence. The performance differences between the two are likely minimal. Do what is easiest to write, to debug, and the most pythonic.
    – user40980
    Commented May 1, 2013 at 22:02

3 Answers 3

1

For Little problems I would suggest storing in memory because memory is cheap and fast and storing in Disk is slow and a costly operation.

32 GB of RAM memory are more common nowadays than solid state drives.

1
  • 1
    I have chosen memory as you answered, as someone in the comments said, more easy and pythonic.
    – gc5
    Commented May 1, 2013 at 22:08
2

I think the answer to this is: it depends.

If it is important that you do not lose this data, then my advice would be to get it onto the disk as soon as possible. Bear in mind that if something bad happens on the computer, anything in memory will be lost.

However if it is more important that your program runs in as performant a way as possible, then writing to memory is bound to be quicker than writing to disk.

1

For problems of this size (few kilobytes to several megabytes), I would say store it in memory, check the length condition, and if it's successful, save it to the disk as a permanent file. Unless the output data retrieved from the command line is very important, it's better to verify the data on the memory. In fact, I would also say that unless the operation performed is "risky" (high chance of data corruption or similar) or there's a noticeable chance of data or power failure, to use the memory first, and then flush it to the disk. RAM is inexpensive and fast but, temporary. Whereas, the hard drive is slow, expensive, but, (more) permanent.

Overall, it's a matter of how big your data is, and how many risks are involved in the process (and you are willing to take).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.