How to be productive around constant 4-6 min interruptions while "slow" code runs?

Question

Not 100% a coding question, more around developer productivity.

I work in Data Science, meaning a lot of my day is spent reading some data, manipulating it, making some charts, training some models, evaluating them, etc...

I'm proud to say the code we are using is very optimized (files are saved in parquet, all manipulations are vectorized and distributed over as many cores as possible, things are memmapped where it would help, regularly used files are cached locally - etc...)

However, there are still dozens of "4min-6min" moments per day where there is nothing to do (the machine is just loading the data from disk, or calculating something...). Sometimes it feels like working with internet connection from the 90's - very hard to stay in flow.

4min is not enough to go do something else without losing one's train of thought, but it is far too long to sit there waiting for the execution to run.

Note:

• This is mainly around ad-hoc analysis and prototyping; we do have scheduled code that runs every day to have all very-slow-running code pre-run before we start the day.
• I've profiled the code, and most of it can't be made any faster.

Any guidance or working patterns you are aware to deal with this?

Answer 1

You cannot argue with physics.

When ad-hoc calculations start to take so long they start to bother you, and you cannot optimize them any more, the only sensible option is to organize the work differently and take those calculations out of the "ad-hoc" procedures into a more planned schedule, like the other code you already have put in there.

That will leave you with the question which ad-hoc work you can still do, since I guess this is an important part of your work. My best recommendation here is to limit the amount of data you use in ad-hoc calculations, and put the larger amounts only into the planned runs. Since you did not come up with this solution by yourself, I guess this may require some work (and especially some thoughtwork) in your case - unfortunately, that is nothing we can really help you with. You may have to develop some tools first to filter and reduce the data in a sensible way, but in my experience, it is often worth to invest some time into that direction.

It does not matter how much you optimize and how much hardware you throw into the ring, when you have enough data to play with and don't restrict systematically how and and how much you use it, you will sooner or later run into the same issue again.

Answer 2

Depending on how often those happen - take a break. Get up, walk around, think, make a drink, something. The truth of the matter is that no human can be focused 100%, and no employee will work 100% of their working hours. That's just not how humans works.

This long running tasks give you the perfect excuse to, if nothing else, rest your eyes. Due to being a smoker, and some quirks in Polish occupational safety laws, I tend to take a short break roughly every hour.

If you're not taking a break, you can find something productive to do regardless - prepare what-ifs depending on the task outcome, read up on stuff related to different tasks, catch up on e-mail. Because my work requires extremely broad knowledge and frequent task switching, I tend to spend those times doing quick, shallow, research into potentially useful stuff. At the very least I will have some names to look further into if the topic actually comes up.

Obligatory relevant XKCD.

Answer 3

I find the “Site Reliability Engineering” book (SRE) has a lot of wisdom around dealing with what that book terms "interrupts work," which is roughly what you're describing, just with the interrupts arising from the same person dealing with them. Relevant chapter: https://sre.google/sre-book/dealing-with-interrupts/

Summary: One option is to reorganize to make the interrupts go away, but if that's not possible it's best to organize around them so as to minimize the disruption. The SRE book's recommendation is to spend no more than 50% of an average workday dealing with interruptions and that the rest of the time should be spent automating those interruptions away.

Answer 4

One way of dealing with 6-7 minute interrupts is by not having them.

When speed matters, programming language matters. A compiled program will be matters of magnitude faster then an interpreted program.

In a lower level language also often there are optimization techniques possible not available in higher level languages.

Answer 5

What about reading up on one of the issues related to the current work? For example:

• Is it possible to optimise the current software, so it runs even more efficiently.
• Are there any different, better algorithms to try out?
• Are there any other or new methods or approaches connected the current domain problem?
• Drafting report or documentation.
• Searching for papers, books etc. For later reading.

In this way you won't go too far from your problem, and will still be productive in some way.