3

Assuming IO is not an issue, is saving intermediate results considered a best practice? What are the pros and cons, and situations that warrant doing so or not?

Say I have two components along a long pipeline,
others--> Component_1 --> Component_2 --> others.

I can either save the output from Component_1, pass the path to Component_2, and have Component_2 read and process from there. Or, I can return output from Component_1, and pass output to Component_2.

Context:
This is for data processing tasks, where the server process itself can run continuously, but each user input data causes a single run through the pipeline, and it completes before it retrieves the next item from the input queue.

Pros of saving:
1. Makes testing and debugging a bit easier? I don't have to save the output from Component_1 in my test/debug code, before doing stuff to it, if I don't want to rerun Component_1. A debugger that saves all intermediate data can do that as well of course, but it saving everything means it might take a while to run.
2. Makes debugging if actual runs fail easier. Same point as previous.

Cons:
1. Performance hit, but we assume it is negligible here.
2. Having to move all intermediate files to trash somewhere during/at the end of runs.
3. Having a separate debug folder with unique ID tag for each run, but that's usually necessary anyway, if only to store output that the UI retrieves and presents.

Improve this question
5
  • 1
    I would save result of Component1 to external resources only in cases where processing in Component2 can be delayed.
    – Fabio
    Commented Jul 25, 2019 at 10:52
  • 2
    You already listed pros and cons, which means that you you're aware that both techniques can be advantageous in specific situations. Best practice is selecting an option that fits the situation. Additional bonus points if you implement it such that you can revert the decision. For example, if your components read from stdin and write to stdout, you can use either intermediate files or pipes.
    – Hans-Martin Mosner
    Commented Jul 25, 2019 at 11:56
  • 1
    Have you considered the possibility of returning while asynchronously saving?
    – Mark Cilia Vincenti
    Commented Jul 25, 2019 at 12:10
  • 1
    The save-to-disc approach also makes it easier to scale out using multiple processes/cpus. However, if there's no need...
    – Erik Eidt
    Commented Jul 25, 2019 at 15:29
  • Thanks guys! I'm new to building real deployed programs so I'm hoping to gain some insights from experienced folks on things I may never have considered. From what I understand so far there's 1. whether Component2 can wait for IO to complete, 2. making it easily convertable, 3. asynchronously saving, and 4. allowing for spawning threads between 1 and 2. I've not thought of most of them actually, and 4 is relevant to my current case. These really helped, thanks!
    – Melvin
    Commented Jul 25, 2019 at 16:43

1 Answer 1

Reset to default
3

Basically it boils down to "How much does your software cost to run?"

Every extra bit of work your code does costs money in terms of CPU, Memory, Disk Space, Bandwidth, electricity, rack space etc etc.

If its low volume, the extra cost will be tiny. But once you scale up you will be spending thousands of pounds every month. If you can knock 10% off the bill at some point you will want to.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.