How to reconcile 'notebook'-style programming with other best practices?

Question

I try to write most of my code using standard practices. Those include amongst other and related to this specific question having short-ish functions, well encapsulated objects when OO is used, not too many function arguments or return values (I mostly write in languages where implicitly returning tuples is allowed, but in C for example I would also try not to return a 100-fields struct containing nearly all the state), and I do not need to be convinced that this is useful for productivity, correctness and knowledge transfer.

However, I have been recently attracted very much by 'notebook'-centered programming, using (in my field) IPython or Matlab LiveScripts. As I work in non-SW related research, those are interesting because you do not often know "what you are looking for" as you are writing code. The advantage of notebooks in that situation is that you can load data once and do a lot of housekeeping, then start experimenting a lot, plot Y in function of X, realize something, plot A in function of X instead to check, and so on and so forth.

The major issue I have with this is that I find 'notebook'-style programming to encourage writing mainly purely procedural code because you need to be able to break anywhere and have all data available. The notion of 'codeblocks' instead of functions also pushes you in that direction.

To be fair, my current approach is not all rosy when it comes to the actual research process: I end up having to rerun the whole code many times because I have to rewrite part of it in order to expose some internal state, or other vector to plot.

Finally, this is not a problem that I feel is easily solved by relying on the debugger. First of all in certain situations the python debugger becomes excruciatingly slow (this might be due to the large amounts of data and processing, but might be tool related: I have never had perf issues with CLI pdb (which I don't often use because of its...minimal interface) but the vscode debugger often grinds down), but more importantly, using a debugger does not entirely solve the issue that if a function returns A, and you realize that you wanted also B, an intermediate value, 10 minutes later and five functions calls later in the code you don't have it available anymore.

Answer 1

Notebooks and interactive development interfaces in general are meant for simple things, working with components you develop alongside. Write your functions to provide you the output you desire and then use the notebook/etc. to test them, do quick runs, get plots and graphs, etc. You can even use those interfaces to prototype your functions and the like, then move to external modules as those grow more complicated.

If you do this style you should probably set your notebook/etc. to reload modules and such as necessary in order to be up to date with any changes you make to your imported code, though.