I try to write most of my code using standard practices. Those include amongst other and related to this specific question having short-ish functions, well encapsulated objects when OO is used, not too many function arguments or return values (I mostly write in languages where implicitly returning tuples is allowed, but in C for example I would also try not to return a 100-fields struct containing nearly all the state), and I do not need to be convinced that this is useful for productivity, correctness and knowledge transfer.

However, I have been recently attracted very much by 'notebook'-centered programming, using (in my field) IPython or Matlab LiveScripts. As I work in non-SW related research, those are interesting because you do not often know "what you are looking for" as you are writing code. The advantage of notebooks in that situation is that you can load data once and do a lot of housekeeping, then start experimenting a lot, plot Y in function of X, realize something, plot A in function of X instead to check, and so on and so forth.

The major issue I have with this is that I find 'notebook'-style programming to encourage writing mainly purely procedural code because you need to be able to break anywhere and have all data available. The notion of 'codeblocks' instead of functions also pushes you in that direction.

To be fair, my current approach is not all rosy when it comes to the actual research process: I end up having to rerun the whole code many times because I have to rewrite part of it in order to expose some internal state, or other vector to plot.

Finally, this is not a problem that I feel is easily solved by relying on the debugger. First of all in certain situations the python debugger becomes excruciatingly slow (this might be due to the large amounts of data and processing, but might be tool related: I have never had perf issues with CLI pdb (which I don't often use because of its...minimal interface) but the vscode debugger often grinds down), but more importantly, using a debugger does not entirely solve the issue that if a function returns A, and you realize that you wanted also B, an intermediate value, 10 minutes later and five functions calls later in the code you don't have it available anymore.

  • This you do not often know "what you are looking for" as you are writing code in some way means bye-bye to requirements. Is it engineering at all?
    – yegodm
    Mar 21, 2018 at 19:22
  • @yegodm What do you mean by that? Do you wonder if the question falls into the scope of this "software engineering" site (I can see an argument here if you are really strict about the definition of software engineering), or do you question the rigour of the process in which case I am afraid you will be disappointed by how research is 99% poking around in the dark...
    – nathdwek
    Mar 21, 2018 at 19:36
  • I mean engineering is driven by requirements, research - by experimenting. They are two different things, nothing common, except using programming languages. Why reconcile then?
    – yegodm
    Mar 21, 2018 at 19:46
  • 1
    @yegodm If I may I think you might have a bit of a skewed, SW-industry centered vision of research. Maybe I am also presenting our research process as too erratic. Just because you don't know exactly how you are going to achieve something does not mean you cannot architecture your solution.
    – nathdwek
    Mar 21, 2018 at 20:54
  • 1
    It is the difference between prototyping and final development. Having worked with "real engineers" quite a bit converting MatLab and other types of code, I can say software developers often scream at the code the real engineers produce. I am used to it and recognize that there is a lot of work to create robust, hard to break, provable code from quickly built prototypes. You have to rewrite from scratch. There is no other way. Mar 22, 2018 at 0:41

1 Answer 1


Notebooks and interactive development interfaces in general are meant for simple things, working with components you develop alongside. Write your functions to provide you the output you desire and then use the notebook/etc. to test them, do quick runs, get plots and graphs, etc. You can even use those interfaces to prototype your functions and the like, then move to external modules as those grow more complicated.

If you do this style you should probably set your notebook/etc. to reload modules and such as necessary in order to be up to date with any changes you make to your imported code, though.

  • 1
    > You can even use those interfaces to prototype your functions and the like, then move to external modules as those grow more complicated. I like to treat these notebooks like one step above a REPL. They're very nice for prototyping.
    – Andrew
    Mar 21, 2018 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.