Programming cleanly when writing scientific code

Question

I don't really write large projects. I'm not maintaining a huge database or dealing with millions of lines of code.

My code is primarily "scripting" type stuff - things to test mathematical functions, or to simulate something - "scientific programming". The longest programs I've worked on up to this point are a couple hundred lines of code, and most of the programs I work on are around 150.

My code is also crap. I realized this the other day as I was trying to find a file I wrote a while ago but that I probably overwrote and that I don't use version control, which is probably making a large number of you cringe in agony at my stupidity.

The style of my code is convoluted and is filled with obsolete comments noting alternate ways to do something or with lines of code copied over. While the variable names always start out very nice and descriptive, as I add or change things as per e.g., something new that someone wants tested, code gets overlayed on top and overwritten and because I feel like this thing should be tested quickly now that I have a framework I start using crappy variable names and the file goes to pot.

In the project I'm working on now, I'm in the phase where all of this is coming back to bite me in a big way. But the problem is (aside from using version control, and making a new file for each new iteration and recording all of it in a text file somewhere, which will probably help the situation dramatically) I don't really know how to proceed with improving my actual coding style.

Is unit testing necessary for writing smaller pieces of code? How about OOP? What sorts of approaches are good for writing good, clean code quickly when doing "scientific programming" as opposed to working on larger projects?

I ask these questions because often, the programming itself isn't super complex. It's more about the math or science that I'm testing or researching with the programming. E.g., is a class necessary when two variables and a function could probably take care of it? (Consider these are also generally situations where the program's speed is preferred to be on the faster end - when you're running 25,000,000+ time steps of a simulation, you kinda want it to be.)

Perhaps this is too broad, and if so, I apologize, but looking at programming books, they often seem to be addressed at larger projects. My code doesn't need OOP, and it's already pretty darn short so it's not like "oh, but the file will be reduced by a thousand lines if we do that!" I want to know how to "start over" and program cleanly on these smaller, faster projects.

I would be glad to provide more specific details, but the more general the advice, the more useful, I think. I am programming in Python 3.

---

Someone suggested a duplicate. Let me make clear I'm not talking about ignoring standard programming standards outright. Clearly, there's a reason those standards exist. But on the other hand, does it really make sense to write code that is say OOP when some standard stuff could have been done, would have been much faster to write, and would have been a similar level of readability because of the shortness of the program?

There's exceptions. Further, there's probably standards for scientific programming beyond just plain standards. I'm asking about those as well. This isn't about if normal coding standards should be ignored when writing scientific code, it's about writing clean scientific code!

---

Update

Just thought I'd add a "not-quite-one-week-later" sort of update. All of your advice was extremely helpful. I now am using version control - git, with git kraken for a graphical interface. It's very easy to use, and has cleaned up my files drastically - no more need for old files sticking around, or old versions of code commented out "just in case".

I also installed pylint and ran it on all of my code. One file got a negative score initially; I'm not even sure how that was possible. My main file started at a score of ~1.83/10 and now is at ~9.1/10. All of the code now conforms fairly well to standards. I also ran over it with my own eyes updating variable names that had gone...uhm...awry, and looking for sections to refactor.

In particular, I asked a recent question on this site on refactoring one of my main functions, and it now is a lot cleaner and a lot shorter: instead of a long, bloated, if/else filled function, it is now less than half the size and much easier to figure out what is going on.

My next step is implementing "unit testing" of sorts. By which I mean a file that I can run on my main file which looks at all the functions in it with assert statements and try/excepts, which is probably not the best way of doing it, and results in a lot of duplicate code, but I'll keep reading and try to figure out how to do it better.

I've also updated significantly the documentation I'd already written, and added supplementary files like an excel spreadsheet, the documentation, and an associated paper to the github repository. It kinda looks like a real programming project now.

So...I guess this is all to say: thank you.

Answer 1

This is a pretty common problem for scientists. I've seen it a lot, and it always stems by the fact that programming is something you pick on the side as a tool to do your job.

So your scripts are a mess. I'm going to go against common sense and say that, assuming you're programming alone, this is not so bad! You're never going to touch most of what you write ever again, so spending too much time to write pretty code instead of producing "value" (so the result of your script) isn't going to do much to you.

However, there is going to be a time where you need to go back to something you did and see exactly how something was working. Additionally, if other scientists will need to review your code, it's really important for it to be as clear and concise as possible, so that everyone can understand it.

Your main problem is going to be readability, so here's a few tips for improving:

Variable names:

Scientists love to use concise notations. All mathematical equations usually use single letters as variables, and I wouldn't be surprised to see lots and lots of very short variables in your code. This hurts readability a lot. When you'll go back to your code you're not going to remember what those y, i and x2 represent, and you'll spend a lot of time trying to figure it out. Try instead naming your variables explicitly, using names that represent exactly what they are.

Split your code into functions:

Now that you renamed all your variables, your equations look terrible, and are multiple lines long.

Instead of leaving it in your main program, move that equation to a different function, and name it accordingly. Now instead of having a huge and messed up line of code, you'll have a short instructions telling you exactly what's going on and what equation you used. This improves both your main program, as you don't even have to look at the actual equation to know what you did, and the equation code itself, as in a separate function you can name your variables however you want, and go back to the more familiar single letters.

On this line of thought, try to find out all the pieces of code that represent something, especially if that something is something you have to do multiple times in your code, and split them out into functions. You'll find out that your code will quickly become easier to read, and that you'll be able to use the same functions without writing more code.

Icing on the cake, if those functions are needed in more of your programs you can just make a library for them, and you'll have them always available.

Global variables:

Back when I was a beginner, I thought this was a great way for passing around data I needed in many points of my program. Turns out there are many other ways to pass around stuff, and the only things global variables do is giving people headaches, since if you go to a random point of your program you'll never know when that value was last used or edited, and tracking it down will be a pain. Try to avoid them whenever possible.

If your functions need to return or modify multiple values, either make a class with those values and pass them down as a parameter, or make the function return multiple values (with named tuples) and assign those values in the caller code.

Version Control

This doesn't directly improve readability, but helps you do all the above. Whenever you do some changes, commit to version control (a local Git repository will be fine enough), and if something doesn't work, look at what you changed or just roll back! This will make refactoring your code way easier, and will be a safety net if you accidentally break stuff.

Keeping all this in mind will allow you to write clear and more effective code, and will also help you find possible mistakes faster, as you won't have to wade through gigantic functions and messy variables.

Answer 2

Physicist here. Been there.

I would argue that your problem is not about the choice of tools or programming paradigms (unit testing, OOP, whatever). It's about the attitude, the mindset. The fact the your variable names are well chosen at first and end up being crap is revealing enough. If you think of your code as “run once, then throw away”, then it will inevitably be a mess. If you think of it as the product of craft and love, it will be beautiful.

I believe there is only one recipe for writing clean code: write it for the human being who is going to read it, not for the interpreter that is going to run it. The interpreter doesn't care if your code is a mess, but the human reader does care.

You are a scientist. You probably can spend a lot of time polishing a scientific article. If your first draft looks convoluted, you will refactor it until the logic just flows in the most natural way. You want your colleagues to read it and find the arguments crystal clear. You want your students to be a able to learn from it.

Writing clean code is exactly the same. Think of your code as a detailed explanation of an algorithm that only incidentally happens to be machine-readable. Imagine you are going to publish it as an article that people will read. You are even going to show it at a conference and walk the audience through it line by line. Now rehearse your presentation. Yes, line by line! Embarrassing, isn't it? So clean up your slides (err... I mean, your code), and rehearse again. Repeat until you are happy with the result.

It would be still better if, after the rehearsals, you can show your code to real people rather than just imaginary people and your future self. Going through it line by line is called a “code walk”, and it's not a silly practice.

Of course, all of this comes at a cost. Writing clean code takes a lot more time than writing throwaway code. Only you can evaluate whether the benefits outweigh the cost for your particular use case.

As for the tools, I said before they are not that important. However, if I had to choose one, I would say version control is the most useful.

Answer 3

Version control is probably going to give you the most bang for your buck. It isn't only for long-term storage, it's great for tracking your short-term experimentation and going back to the last version that worked, keeping notes along the way.

Next most useful are unit tests. The thing about unit tests is even code bases with millions of lines of code are unit tested one function at a time. Unit testing is done in the small, at the lowest level of abstraction. That means there is fundamentally no difference between unit tests written for small code bases and those used for large. There is just more of them.

Unit tests are the best way to keep from breaking something that was already working when you fix something else, or at least to tell you quickly when you do. They are actually more useful when you are as not as skilled a programmer, or do not know how or do not want to write more verbose code that is structured to make errors less likely or more obvious.

Between version control and writing unit tests as you go, you code will naturally become a lot cleaner. Other techniques for cleaner coding can be learned when you hit a plateau.

Answer 4

(aside from using version control, and making a new file for each new iteration and recording all of it in a text file somewhere, which will probably help the situation dramatically)

You would probably have figured this out yourself, but if you have to "record all of it in a text file somewhere" you're not using the version control system to its full potential. Use something like Subversion, git, or Mercurial and write a good commit message with every commit and you will have a log which serves the purpose of the text file but can't get separated from the repository.

That aside, using version control is the most important thing you can do for a reason which none of the existing answers mentions: reproducibility of results. If you either can use your log messages or add a note to the results with the revision number then you can be sure of being able to regenerate the results, and you will be better placed to publish the code with the paper.

Is unit testing necessary for writing smaller pieces of code? How about OOP? What sorts of approaches are good for writing good, clean code quickly when doing "scientific programming" as opposed to working on larger projects?

Unit testing is never necessary, but it is useful if (a) the code is modular enough that you can test units rather than the whole thing; (b) you can create tests. Ideally you would be able to write the expected output by hand rather than generate it with the code, although generating it by code can at least give you regression tests which tell you whether something changed its behaviour. Just consider whether the tests are more likely to be buggy than the code they're testing.

OOP is a tool. Use it if it helps, but it's not the only paradigm. I assume that you only really know procedural programming: if that's the case, then in the context described I think you would benefit more from studying functional programming than OOP, and in particular the discipline of avoiding side-effects where possible. Python can be written in a very functional style.

Answer 5

In graduate school, I wrote some algorithm-heavy code myself. It's a bit of a tough nut to crack. To put it coarsely, a lot of programming conventions are built around the idea of putting information into a database, retrieving it at the right time, and then massaging that data to present it to a user, typically using a library for any math- or algorithm-heavy parts of that process. For these programs, everything you've heard about OOP, breaking code into short functions, and making everything easily understandable at a glance where possible is excellent advice. But it doesn't quite work for algorithm-heavy code, or code that implements complex mathematical calculations and little else.

If you're writing scripts to perform scientific calculations, you probably have papers with the equations or algorithms you use written in them. If you're using new ideas you've discovered on your own, you're hopefully going to publish them in papers of your own. In this case, the rule is: You want your code to read as much like the published equations as possible. Here is an answer on Software Engineering.SE with over 200 upvotes advocating for this approach and explaining what it looks like: Is there an excuse for short variable names?

As another example, there are some great snippets of code in Simbody, a physics simulation tool used for physics research and engineering. These snippets have a comment showing an equation being used for a calculation, followed by code that reads as closely to the equations being implemented as possible.

ContactGeometry.cpp:

// t = (-b +/- sqrt(b^2-4ac)) / 2a
// Discriminant must be nonnegative for real surfaces
// but could be slightly negative due to numerical noise.
Real sqrtd = std::sqrt(std::max(B*B - 4*A*C, Real(0)));
Vec2 t = Vec2(sqrtd - B, -sqrtd - B) / (2*A);

ContactGeometry_Sphere.cpp:

// Solve the scalar Jacobi equation
//
//        j''(s) + K(s)*j(s) = 0 ,                                     (1)
//
// where K is the Gaussian curvature and (.)' := d(.)/ds denotes differentiation
// with respect to the arc length s. Then, j is the directional sensitivity and
// we obtain the corresponding variational vector field by multiplying b*j. For
// a sphere, K = R^(-2) and the solution of equation (1) becomes
//
//        j  = R * sin(1/R * s)                                        (2)
//          j' =     cos(1/R * s) ,                                      (3)
//
// where equation (2) is the standard solution of a non-damped oscillator. Its
// period is 2*pi*R and its amplitude is R.

// Forward directional sensitivity from P to Q
Vec2 jPQ(R*sin(k * s), cos(k * s));
geod.addDirectionalSensitivityPtoQ(jPQ);

// Backwards directional sensitivity from Q to P
Vec2 jQP(R*sin(k * (L-s)), cos(k * (L-s)));
geod.addDirectionalSensitivityQtoP(jQP);

Answer 6

So, my day job is in research data publication and preservation for the University of California system. A couple of folks have mentioned reproducibility, and I think that's really the core issue here: documenting your code the way you'd document anything else someone needs to reproduce your experiment, and, ideally, writing code that makes it straightforward for someone else both to reproduce your experiment and to check your results for sources of error.

But something that I haven't seen mentioned, that I think is important, is that funding agencies are increasingly looking at software publication as a part of data publication, and at making software publication a requirement for open science.

To that end, if you want something specific, targeted at researchers rather than general software developers, I can't recommend the Software Carpentry organization highly enough. If you can attend one of their workshops, great; if all you have time/access to do is read some of their papers on scientific computing best practices, that's good too. From the latter:

Scientists typically develop their own software for these purposes because doing so requires substantial domain-specific knowledge. As a result, recent studies have found that scientists typically spend 30% or more of their time developing software. However, 90% or more of them are primarily self-taught, and therefore lack exposure to basic software development practices such as writing maintainable code, using version control and issue trackers, code reviews, unit testing, and task automation.

We believe that software is just another kind of experimental apparatus and should be built, checked, and used as carefully as any physical apparatus. However, while most scientists are careful to validate their laboratory and field equipment, most do not know how reliable their software is. This can lead to serious errors impacting the central conclusions of published research. …

In addition, because software is often used for more than a single project, and is often reused by other scientists, computing errors can have disproportionate impacts on the scientific process. This type of cascading impact caused several prominent retractions when an error from another group's code was not discovered until after publication.

A high-level outline of the practices they recommend:

1. Write programs for people, not computers
2. Let the computer do the work
3. Make incremental changes
4. Don't repeat yourself (or others)
5. Plan for mistakes
6. Optimize software only after it works correctly
7. Document design and purpose, not mechanics
8. Collaborate

The paper goes into considerable detail on each of these points.

Answer 7

Does it really make sense to write code that is say OOP when some standard stuff could have been done, would have been much faster to write, and would have been a similar level of readability because of the shortness of the program?

Personal answer:
I do a lot of scripting for scientific purposes too. For smaller scripts, I simply try to follow general good programming practices (i.e. using version control, practicing self-control with variable names). If I'm just writing something to quickly open or visualize a dataset, I don't bother with OOP.

General Answer:
"It depends." But if you are struggling to figure out when to use a programming concept or paradigms, here are a couple things to think about:

• Scalability: Is the script going to stand alone, or will it eventually be used in a larger program? If so, is the larger programming using OOP? Can the code in your script be easily integrated into the larger program?
• Modularity: In general, your code should be modular. However, OOP breaks code into chunks in a very special way. Does that type of modularity (i.e. breaking up your script into classes) make sense for what you are doing?

I want to know how to "start over" and program cleanly on these smaller, faster projects.

#1: Get familiar with what's out there:
Even though you're "just" scripting (and you really just care about the science component), you should take some time to learn about different programming concepts and paradigms. That way, you can have a better idea of what you should/ should not want to use and when. That may sound a bit daunting. And you may still have the question, "Where do I start/ what do I start looking at?" I try to explain a good starting point in the next two bullet points.

#2: Start fixing what you know is wrong:
Personally, I would start with the things that I know are wrong. Get some version control and start to discipline yourself to get better with those variable names (it is a serious struggle). Fixing what you know is wrong may sound obvious. However, in my experience, I've found that fixing one thing leads me to something else, and so on. Before I know it, I've unveiled 10 different things I was doing wrong and figured out how to fix them or how to implement them in a clean way.

#3: Get a programming partner:
If "starting over" for you doesn't involve taking formal classes, consider teaming up with a developer and asking them to review your code. Even if they don't understand the science part of what you're doing, they might be able to tell you what you could have done to make your code more elegant.

#4: Look for consortiums:
I don't know what scientific area you are in. But depending on what you do in the scientific world, try looking for consortiums, working groups, or conference participants. Then see if there are any standards that they are working on. That may lead you to some coding standards. For example, I do a lot of geospatial work. Looking at conference papers and working groups led me to the Open Geospatial Consortium. One of the things they do is work on standards for geospatial development.

I hope that helps!

---

Side note: I know you just used OOP as an example. I didn't want you to think that I got stuck on just how to handle writing code using OOP. It was just easier to write an answer continuing with that example.

Answer 8

I'd recommend to stick to the Unix principle: Keep It Simple, Stupid! (KISS)

Or, put another way: Do one thing at a time, and do it well.

What does that mean? Well, first of all, it means your functions should be short. Any function that cannot be fully understood in purpose, usage, and implementation within a few seconds is definitely too long. It's likely doing several things at once, each of which should be a function of their own. So split it.

In terms of code lines, my heuristic is that 10 lines are a good function, and anything beyond 20 is most likely crap. Other people have other heuristics. The important part is to keep the length down to something you can actually grasp in an instant.

How do you split a long function? Well, first you look for repeating patterns of code. Then you factor out these code patterns, give them a descriptive name, and watch your code shrink. Really, the best refactoring is the refactoring that reduces code size.

This is especially true when the function in question has been programmed with copy-paste. Whenever you see such a repeated pattern, you instantly know that this should likely be turned into a function of its own. This is the principle of Don't Repeat Yourself (DRY). Whenever you are hitting copy-paste, you are doing something wrong! Create a function instead.

Anecdote
I once spent several months refactoring code that had functions of about 500 lines each. After I was done, the total code was about a thousand lines shorter; I had produced negative output in terms of lines of code. I owed the company (http://www.geekherocomic.com/2008/10/09/programmers-salary-policy/index.html). Still, I firmly believe that this was one of my most valuable works I ever did...

Some functions may be long because they are doing several distinct things one after another. These are not DRY violations, but they can also be split. The result is frequently a high level function that calls a handfull of functions that implement the individual steps of the original functions. This will generally increase code size, but the added function names work wonders in making the code more readable. Because now you have a top-level function with all its steps explicitly named. Also, after this split it's clear, which step operates on which data. (Function arguments. You do not use global variables, do you?)

A good heuristic for this kind of sectional function split is whenever you are tempted to write a section comment, or when you find a section comment in your code. This is very likely one of the points where your function should be split. The section comment can also serve to inspire a name for the new function.

The KISS and DRY principles can take you a long way. You do not need to start off with OOP etc. immediately, oftentimes you can achieve great simplifications by just applying these two. However, it does pay off in the long run to know about OOP and other paradigms because they give you additional tools that you can use to make your program code more clear.

Finally, record every action with a commit. You factor something out into a new function, that's a commit. You fuse two functions into one, because they really do the same thing, that's a commit. If you rename a variable, that's a commit. Commit frequently. If a day goes by, and you didn't commit, you likely did something wrong.

Answer 9

I agree with the others that version control will solve many of your problems right away. Specifically:

• No need to maintain a list of what changes have been made, or having lots of copies of a file, etc. since that's what version control takes care of.
• No more files lost due to overwrites, etc. (as long as you just stick to the basics; e.g. avoid "rewriting history")
• No need for obsolete comments, dead code, etc. being kept around "just in case"; once they're committed to version control, feel free to nuke them. This can feel very liberating!

I would say don't overthink it: just use git. Stick to simple commands (e.g. just a single master branch), perhaps use a GUI, and you should be fine. As a bonus you can use gitlab, github, etc. for free publishing and backups ;)

The reason I wrote this answer was to address two things you might try that I haven't seen mentioned above. The first is to use assertions as a lightweight alternative to unit testing. Unit tests tend to sit "outside" the function/module/whatever being tested: they typically send some data into a function, receive a result back, then check some properties of that result. This is generally a good idea, but might be inconvenient (especially for "throw away" code) for a few reasons:

• Unit tests need to decide what data they'll give to the function. That data must be realistic (otherwise there's little point testing it), it must have the correct format, etc.
• Unit tests must have "access" to the things they want to assert. In particular, unit tests can't check any of the intermediate data inside a function; we would have to break apart that function into smaller pieces, test those pieces, and plug them together somewhere else.
• Unit tests are also assumed to be relevant to the program. For example, test suites can become "stale" if there have been large changes since they were last run, and there might even be a bunch of tests for code that's not even used anymore.

Assertions don't have these drawbacks, since they're checked during normal execution of a program. In particular:

• Since they're run as part of the normal program execution, we have actual real-world data to play with. This doesn't require separate curation and (by definition) is realistic and has the correct format.
• Assertions can be written anywhere in the code, so we can put them wherever we have access to the data we want to check. If we want to test some intermediate value in a function, we can just put some assertions into the middle of that function!
• Since they're written in-line, assertions can't get "out of sync" with the structure of the code. If we make sure that assertions are checked by default, we also don't need to worry about them getting "stale" since we will see immediately whether or not they pass the next time we run the program!

You mention speed as a factor, in which case assertion checking might be undesirable in that loop (but still useful for checking setup and subsequent processing). However, almost all implementations of assertions provide a way to turn them off; for example in Python they can apparently be disabled by running with the -O option (I didn't know this, since I've never felt a need to disable any of my assertions before). I would recommend that you leave them on by default; if your coding/debugging/testing cycle slows down you might be better off testing with a smaller subset of your data, or performing fewer iterations of some simulation during testing, or whatever. If you do end up disabling assertions in non-test runs for performance reasons, the first thing I recommend you do is measure if they're actually the source of the slowdown! (It's very easy to delude ourselves when it comes to performance bottlenecks)

My final piece of advice would be to use a build system which manages your dependencies. I personally use Nix for this, but have heard good things about Guix as well. There are also alternatives like Docker, which are far less useful from a scientific perspective but perhaps a little more familiar.

Systems like Nix are only recently becoming (a little) popular, and some might consider them overkill for "throw away" code like you describe, but their benefit for reproducibility of scientific computing is enormous. Consider a shell script for running an experiment, like this (e.g. run.sh):

#!/usr/bin/env bash
set -e
make all
./analyse < ./dataset > output.csv

We can rewrite it into a Nix "derivation" instead, like this (e.g. run.nix):

with import <nixpkgs> {};
runCommand "output.csv" {} ''
  cp -a ${./.} src
  cd src
  make all
  ./analyse < ./dataset > $out
''

The stuff between ''...'' is bash code, the same as we had before, except that ${...} can be used to "splice" in the content of other strings (in this case ./., which will expand to the path of the directory containing run.nix). The with import ... line imports Nix's standard library, which provides runCommand for running bash code. We can run our experiment using nix-build run.nix, which will give out a path like /nix/store/1wv437qdjg6j171gjanj5fvg5kxc828p-output.csv.

So what does this buy us? Nix will automatically set up a "clean" environment, which only has access to things that we've explicitly asked for. In particular it doesn't have access to variables like $HOME or any of the system software we have installed. This makes the result independent of the details of our current machine, like the contents of ~/.config or the versions of programs we happen to have installed; AKA the stuff that prevents other people from replicating our results! This is why I added that cp command, since the project won't be accessible by default. It might seem annoying that the system's sofware isn't available to a Nix script, but it goes the other way too: we don't need anything installed on our system (other than Nix) to make use of it in a script; we just ask for it and Nix will go off and fetch/compile/whatever is necessary (most things will be downloaded as binaries; the standard library is also huge!). For example, if we want a bunch of particular Python and Haskell packages, for some particular versions of those languages, plus some other junk (because why not?):

with import <nixpkgs> {};
runCommand "output.csv"
  {
    buildInputs = [
      gcc49 libjson zlib
      haskell.packages.ghc802.pandoc
      (python34.withPackages (pyPkgs: [
        pyPkgs.beautifulsoup4 pyPkgs.numpy pyPkgs.scipy
        pyPkgs.tensorflowWithoutCuda
      ]))
    ];
  }
  ''
    cp -a ${./.} src
    cd src
    make all
    ./analyse < ./dataset > $out
  ''

The same nix-build run.nix will run this, fetching everything we asked for first (and caching it all in case we want it later). The output (any file/directory called $out) will be stored by Nix, which is the path it spits out. It's identified by the cryptographic hash of all the inputs we asked for (script contents, other packages, names, compiler flags, etc.); those other packages are identified by hashes of their inputs, and so on such that we have a full chain of provinence for everything, right back to the version of GCC that compiled the version of GCC that compiled bash, and so on!

Hopefully I've shown that this buys us a lot for scientific code, and is reasonably easy to get started with. It's also starting to get taken very seriously by scientists, e.g. (top Google hit) https://dl.acm.org/citation.cfm?id=2830172 so might be a valuable skill to cultivate (just like programming)

Answer 10

Without going to the full-fledge version control + packaging + unit tests kind of mindset (which are good programming practices that you should try to achieve at some point), one intermediate solution that I think would fit is to use Jupiter Notebook. This seems to better integrate with scientific computation.

It has the advantage that you can mix your thoughts with the code; explaining why an approach is better than another and leaving old code as-is in an ad-hoc section. Besides using cells properly will naturally lead you to fragment your code and organize it into functions which can help its understanding.

Answer 11

The top answers are already good, but I wanted to address some of your questions directly.

Is unit testing necessary for writing smaller pieces of code?

The size of the code is not directly related to the need for unit tests. It is related indirectly: unit tests are more valuable in complex codebases, and small codebases are generally not as complex as larger ones.

Unit tests shine for code where it's easy to make mistakes, or when you're going to have many implementations of this code. Unit tests do little to help you with current development, but they do a lot to prevent you making mistakes in the future that cause existing code to suddenly misbehave (even though you didn't touch that thing).

Let's say you have an application where Library A performs the squaring of numbers, and Library B applies the Pythagorean theorem. Obviously, B depends on A. You need to fix something in library A, and let's say you introduce a bug that cubes numbers instead of squaring them.

Library B will suddenly start misbehaving, possibly throwing exceptions or simply giving wrong output. And when you look at the version history of library B, you see that it is untouched. The problematic end result is that you have no indication of what could be going wrong, and you're going to have to debug the behavior of B before you realize the problem is in A. That's wasted effort.

Enter unit tests. These tests confirm that library A is working as intended. If you introduce a bug in library A that causes it to return bad results, then your unit tests will catch that. Therefore, you won't be stuck trying to debug library B.
This is beyond your scope, but in a continuous integration development, unit tests are executed whenever someone commits some code, which means you'll know you broke something ASAP.

Especially for complicated mathematical operations, unit tests can be a blessing. You do a few example calculations, and then you write unit tests which compared your calculated output and your actual output (based on the same input parameters).

However, note that unit tests won't help you create good code, but rather maintain it. If you usually write code once and never revisit it, unit tests are going to be less beneficial.

How about OOP?

OOP is a way of thinking about distinct entities, for example:

When a Customer wants to purchase a Product, he talks to the Vendor to receive an Order. The Accountant will then pay the Vendor.

Compare this to how a functional programmer thinks about things:

When a customer wants to purchaseProduct(), he talktoVendor() so they will sendOrder() to him. The acccountant will then payVendor().

Apples and oranges. Neither of them is objectively better than the other. One interesting thing to note is that for OOP, Vendor is mentioned twice but it refers to the same thing. However, for functional programming, talktoVendor() and payVendor() are two separate things.
This showcases the difference between the approaches. If there is a lot of shared vendor-specific logic between these two actions, then OOP help reduce code duplication. However, if there is no shared logic between the two, then merging them into a single Vendor is futile work (and therefore fuctional programming is more efficient).

More often than not, mathematical and scientific calculations are distinct operations that do not rely on implicit shared logic/formulae. Because of that, functional programming is more often used than OOP.

What sorts of approaches are good for writing good, clean code quickly when doing "scientific programming" as opposed to working on larger projects?

Your question implies that the definition of "good, clean code" changes whether you're doing scientific programming or working on larger (I assume you mean enterprise) projects.

The definition of good code does not change. The need to avoid complexity (which can be done by writing clean code), however, does change.

The same argument comes back here.

• If you never revisit old code and fully understand the logic without needing to compartmentalize it, then don't spend excessive effort to make things maintainable.
• If you do revisit old code, or the required logic is too complex for you to tackle all at once (thus requiring you to compartmentalize the solutions), then focus on writing clean, reusable close.

I ask these questions because often, the programming itself isn't super complex. It's more about the math or science that I'm testing or researching with the programming.

I get the distinction you're making here, but when you look back at existing code, you are looking at both the math and the programming. If either is contrived or complex, then you'll struggle to read it.

E.g., is a class necessary when two variables and a function could probably take care of it?

OOP principles aside, the main reason I write classes to house a few data values is because it simplifies declaring method parameters and return values. For example, if I have a lot of methods that use a location (lat/lon pair), then I will quickly tire of having to type float latitude, float longitude and will much prefer to write Location loc.

This is compounded further when you consider that methods generally return one value (unless language specific features exist to return more values), and things like a location would want you to return two values (lat + lon). This incentivizes you to create a Location class to simplify your code.

E.g., is a class necessary when two variables and a function could probably take care of it?

Another interesting thing to note is that you can use OOP without mixing data values and methods. Not every developer agrees here (some call it an antipattern), but you can have anemic data models where you have separate data classes (stores value fields) and logic classes (stores methods).
This is, of course, on a spectrum. You don't need to be perfectly anemic, yu can use it when you consider it appropriate.

For example, a method that simply concatenates the first and last name of a person can still be housed in the Person class itself, because it's not really "logic" but rather a calculated value.

(Consider these are also generally situations where the program's speed is preferred to be on the faster end - when you're running 25,000,000+ time steps of a simulation, you kinda want it to be.)

A class is always as big as the sum of its fields. Taking the example of Location again, which consists of two float values, it's important to note here that a single Location object will take up as much memory as two separate float values.

In that sense, it doesn't matter whether you're using OOP or not. The memory footprint is the same.

Performance itself is also not a big hurdle to cross. The difference between e.g. using a global method or a class method has nothing to do with runtime performance, but has everything to do with compile-time generation of bytecode.

Think of it this way: whether I write my cake recipe in English or Spanish doesn't change the fact that the cake will take 30 minutes to bake (= runtime performance). The only thing that the recipe's language changes is how the cook mixes the ingredients (= compiling the bytecode).

For Python specifically, you don't need to explicitly pre-compile the code before calling it. However, when you don't pre-compile, the compilation will occur when trying to execute the code. When I say "runtime", I mean the execution itself, not the compilation which could preceed the execution.

Answer 12

The tools of the trade are usually invented to solve a need. If you have the need you use the tool, if not, you most likely don't have to.

Specifically, scientific programs are not the end target, they are the means. You write the program to solve a problem you have now - you don't expect that program to be used by others (and having to be maintained) in ten years. That alone mean that you don't have to think into any of the tools that allow the current developer to record history for others like version control, or capture functionality in code like unit tests.

What would benefit you then?

• version control is nice because it allows you to very easily backup your work. As of 2018 github is a very popular place to do so (and you can always move it later if needed - git is very flexible). A cheap and simple substitute for backups are the automatic backup procedures in your operating system (Time Machine for Mac, rsync for Linux etc). Your code needs to be in multiple places!
• Unit tests are nice because if you write them first you are forced to think about how to check what the code actually does, which helps you design a more useful API for your code. This is helpful if you ever get into writing code to be reused later and helps you while changing an algorithm because you know it works on these cases.
• Documentation. Learn to write proper documentation in the programming language you use (javadoc for Java for instance). Write for the future you. In this process you will find that good variable names makes documenting easier. Iterate. Give the same amount of care to your documentation as a poet does to poems.
• Use good tools. Find an IDE that helps you and learn it well. Refactoring like renaming variables to a better name is much easier this way.
• If you have peers consider using peer review. Having an outsider look at and understand your code, is the here-and-now version of the future you you write for. If your peer does not understand your code, you probably won't either later.

Answer 13

Benefits of clean scientific code

• ... looking at programming books, they often seem to be addressed at larger projects.

• ... does it really make sense to write code that is say OOP when some standard stuff could have been done, would have been much faster to write, and would have been a similar level of readability because of the shortness of the program?

It might be helpful to consider your code from the perspective of a future coder.

• Why did they open this file?
• What are they looking for?

From my experience,

Clean code should make it easy to verify your results

• Make it easy for users to know exactly what they need to do to run your program.

• You may want to divide up your program so that individual algorithms can be benchmarked separately.

• Avoid writing functions with counter-intuitive side effects where one unrelated operation causes another operation to behave differently. If you can't avoid it, document what your code needs and how to set it up.

Clean code can serve as example code for future coders

Clear comments (including ones that show how functions should be called) and well separated functions can make a huge difference in how long it takes for somebody just starting out (or future you) to make something useful out of your work.

In addition to this, making a real "API" for your algorithm can make you better prepared if you decide to make your scripts into a real library for somebody else to use.

Recommendations

"Cite" mathematical formulas using comments.

• Add comments to "cite" mathematical formulas, especially if you used optimizations (trig identities, Taylor series, etc).
• If you got the formula from the book, add a comment saying John Smith Method from Some Book 1st Ed. Section 1.2.3 Pg 180, if you found the formula on a website or in a paper, cite that as well.
• I'd recommend avoiding "link only" comments, make sure you refer to the method by name somewhere to allow people to google it, I have run into some "link only" comments that redirected to old internal pages and they can be very frustrating.
• You can attempt to type out the formula in your comment if it's still easy to read in Unicode / ASCII, but this can get very awkward (code comments aren't LaTeX).

Use comments wisely

If you can improve the readability of your code by using good variable names / function names, do that first. Remember that comments will stick around forever until you remove them, so try to make comments that won't go out of date.

Use descriptive variable names

• Single letter variables may be the best option if they are part of a formula.
• It may be crucial for future readers to be able to look at the code you wrote and compare it to the equation you are implementing.
• When appropriate, consider adding a suffix to it to describe its real meaning, e.g,. xBar_AverageVelocity
• As mentioned before, I recommend clearly indicating the formula / method you are using by name in a comment somewhere.

Write code to run your program against known good and known bad data.

Is unit testing necessary for writing smaller pieces of code?

I think unit testing can be helpful, I think the best form of unit testing for scientific code is a series of tests that run on known bad and good data.

Write some code to run your algorithm and check to see how far the result deviates from what you expect. This will help you find (potentially very bad and hard to find) problems where you accidentally hard code something causing a false positive result, or make a mistake that causes the function to always return the same value.

Note that this can be done at any level of abstraction. For example you could test an entire pattern matching algorithm, or you can test a function that just calculates the distance between two results in your optimization process. Start with the areas that are the most crucial to your results first, and/or the parts of the code that you are the most concerned about.

Make it easy to add new test cases, consider adding "helper" functions, and structure your input data effectively. This may mean possibly saving input data to file so that you can easily re-run tests, though be very careful to avoid false positives or biased / trivially solved test cases.

Consider using something like cross validation, see this post on cross validated for more information.

Use version control

I would recommend using version control and hosting your repository on an external site. There are sites that will host repositories for free.

Advantages:

1. It provides a backup in case your hard disk fails
2. It provides a history, which keeps you from worrying if a recent problem that came up was caused by you accidentally changing a file, among other benefits.
3. It allows you to use branching, which is a good way to work on long term / experimental code without affecting unrelated work.

Use caution when copy/pasting code

The style of my code is convoluted and is filled with obsolete comments noting alternate ways to do something or with lines of code copied over.

• Copy/pasting code can save you time, but it's one of the most dangerous things you can do, especially if it's code you didn't write yourself (e.g., if it's code from a colleague).

• As soon as you get the code working and tested, I'd recommend going through it very carefully to rename any variables or comment anything you don't understand.

Answer 14

In addition to the good advice already here, you might want to consider the purpose of your programming, and therefore what is important to you.

"It's more about the math or science that I'm testing or researching with the programming."

If the purpose is to experiment and test something for your own understanding and you know what the results should be then your code is basically a quick throw-away and your current approach may suffice, although could be improved. If the results are not as expected, you can go back and review.

However, if the results of your coding are informing the direction of your research and you don't know what the results should be, then correctness becomes particularly important. An error in your code could lead you draw the wrong conclusions from your experiment with a variety of bad implications for your overall research.

In that case, breaking your code into easily understandable and verifiable functions with unit tests will give you more solid building bricks giving you more confidence in your results and may save you from much frustration later.

Answer 15

As great as version control and unit testing are for keeping your overall code organised and functional, neither actually help you write cleaner code.

• Version control will allow you to see how and when the code got as messy as it is.
• Unit tests will make sure that, despite the code being a complete mess, it still works.

If you want to stop yourself from writing messy code, you need a tool that works where the messes happen: when you're writing the code. A popular kind of tool that does is called a linter. I'm not a python developer, but it looks like Pylint might be a good option.

A linter looks at the code you've written, and compares it to a configurable set of best practices. If the linter has a rule that variables must be camelCase, and you write one in snake_case, it will flag that as a mistake. Good linters have rules ranging from "declared variables must be used" to "The cyclomatic complexity of functions must be less than 3".

Most code editors can be configured to run a linter every time you save, or just generally as you type, and indicate problems inline. If you type something like x = 7, the x will be highlighted, with an instruction to use a longer, better name (if that's what you have configured). This works like spellcheck in most word processors, making it hard to ignore, and helping to build better habits.

Answer 16

Everything you listed is a tool in the metaphorical toolbox. Like anything in life, different tools are appropriate for different tasks.

Compared to other engineering fields, software works with a bunch of individual pieces that by themselves, are pretty simple. An assignment statement doesn't evaluate differently depending on temperature fluctuations of the room. An if statement doesn't corrode into place and keep returning the same thing after awhile. But because the individual elements are so simple, and software authored by humans, those elements are combined into larger and larger pieces until the result becomes so large and complex it reaches the limits of what people can mentally manage.

As software projects have grown and grown, people have studied them and created tools to try to manage that complexity. OOP is one example. More and more abstract programming languages are another means. Because much of the money in software is doing more more more, tools to achieve that are what you are going to see and read about. But it seems those situations don't apply to you.

So, don't feel like you need to be doing any of that. At the end of the day, the code is just a means to an end. Unfortunately, what will best give you the right perspective on what is and what isn't appropriate is to work on some larger projects, as it is a lot harder to know what is missing when the toolbox is your mind.

In any case, I wouldn't worry about not using OOP or other techniques as long as your scripts are small. Many of the problems you described are just general professional organizational skills, i.e. not losing an old file is something that all fields have to deal with.

Answer 17

In addition to all the good suggestions provided so far, one practice I learned over time and find essential is to very liberally add detailed commentary to your code. It is the single most important thing for me when I come back to something after a long lapse of time. Explain to yourself what you're thinking. It takes a little time to do but it's relatively easy and mostly painless.

I sometimes have two or three times as many lines of comments as I do of code, especially when the concepts or techniques are new to me and bear explaining to myself.

Do version control, improve your practices, etc. .... all of the above. But explain things to yourself as you go. It works really well.

Answer 18

What qualities are important for this kind of program?

It probably doesn't matter whether its easy to maintain or evolve it, because the chances are that's not going to happen.

It probably doesn't matter how efficient it is.

It probably doesn't matter whether it has a great user interface or whether it's secure against malicious attackers.

It may matter that it's readable: that someone reading your code can easily convince themselves that it does what it claims to do.

It certainly matters that it's correct. If the program gives incorrect results, that's your scientific conclusions out of the window. But it only needs to process correctly the input you are actually asking it to process; it really doesn't matter very much whether it falls over if given negative input data values, if all your data values are positive.

It also matters that you maintain some level of change control. Your scientific results need to be reproducible, and that means that you need to know which version of the program produced the results you intend to publish. Because there's only one developer, the change control doesn't need to be very elaborate, but you need to make sure that you can go back to a point in time and reproduce your results.

So don't worry about programming paradigms, object orientation, algorithmic elegance. Do worry about clarity and readability and about traceability of your changes over time. Don't worry about the user interface. Don't worry about testing every possible combination of input parameters, but do enough testing to be confident (and to make others confident) that your results and conclusions are valid.

Answer 19

I've worked in a similar environment with academics who write a lot of (math/science) code but their progress is slow due to the same reasons that you describe. However I've noticed one particular thing that went well which I think can also help you: build up and maintain a collection of specialized libraries which can be used across multiple projects. These libraries should provide utility functions and will therefore help keep your current project specific to the problem domain.

For example, you might have to deal with a lot of co-ordinate transformations in your field (ECEF, NED, lat/lon, WGS84 etc.), so a function like convert_ecef_to_ned() should go into a new project called CoordinateTransformations. Put the project under version control and host it on your department's servers so that other people can use (and hopefully improve) it. Then after a few years you should have a robust collection of libraries with your projects only containing code specific to a particular problem/research domain.

Some more general advice:

• Always aim to model your particular problem as accurately as possible, no matter what it is. That way the software design questions such as the what/where/how to put a variable should become much more obvious to answer.
• I wouldn't bother with test driven development since scientific code describe ideas and concepts and is more creative and fluid; there are no APIs to define, services to maintain, risks to other people's code when changing functionality etc.

Answer 20

The following are my opinions and very much influenced by my own particular path.

Coding often engenders dogmatic perspectives in how you should do things. Instead of techniques & tools, I think you need to look at the cumulative values & costs to decide on an appropriate strategy.

Writing good, readable, debuggable, solid code takes a lot of time & effort. In many cases, given a limited planning horizon, it is not worthwhile doing this (analysis paralysis).

One colleague had a rule of thumb; if you are doing essentially the same sort of thing for the third time then invest effort, otherwise a quick & dirty job is appropriate.

Testing of some sort is essential, but for one off projects, simply eyeballing may be sufficient. For anything substantial, tests & test infrastructure are essential. The value is it frees you when coding, the cost is that if the test focuses on a particular implementation then the tests need maintenance too. Tests also remind you of how things are are supposes to operate.

For my own one off scripts (often for things like validating an estimate of a probability, or similar), I found two small things very useful: 1. Include a comment showing how it the code is used. 2. Include a brief description of why you wrote the code. These things are awfully obvious when you write the code, but obviousness wastes with time :-).

OOP is about reusing code, abstracting, encapsulating, factoring, etc. Very useful, but easy to get lost in if producing quality code & design is not your end goal. It takes time & effort to produce quality stuff.

Answer 21

While I think that unit tests have their merits, they are of doubtful value for scientific development - they are often just too small to offer lots of value.

But I really like integration tests for scientific code:

Isolate a small chunk of your code which could work on it's own, e.g. the ETL pipeline. Then write a test which provides the data, run the etl pipeline (or just a step) and then test that the result matches your expectations. While the tested chunk can be lots of code, the test provides still value:

1. You have a convienient hook to re-execute your code, which helpts to run it often.
2. You can test some assumptions in your test
3. If somethink breaks, it is easy to add a failing test and make the fix
4. You codify the expected inputs / outputs, avoiding the usual headache which results from trying to guess the input data format.
5. While not as lean as unit tests, IT-tests still help to break your code apart and force you to add some boundaries in your code.

I am using this technique often, and often end up with a relativly readable main function but sub-functions are often quite long and ugly, but can be modified and rearranged quickly due to robust I/O boundaries.

Answer 22

I normally work on a very large source base. We use all the tools you mention. Recently, I've started working on some python scripts for a side project. They are a few dozen to a few hundred lines at most. Out of habit, I committed my scripts to source control. This has been useful because I can create branches for trying out experiments that might not work. I can fork if I need to duplicate the code and modify it for another purpose. This leaves the original in tact in case I need to bring it out again.

For "unit tests" I just have some input files that are intended to produce some known output that I check by hand. I could probably automate it, but it feels like it would take more time to do that than I would save by doing it. It probably depends on how often I have to modify and run the scripts. Either way, if it works, do it. If it's more trouble than it's worth, don't waste your time.

Answer 23

With writing code - as with writing in general - the main question is:

Which reader do you have in mind? or Who consumes your code?

Things like formal coding guidelines make no sense when you are your sole audience.

That being said, on the other hand, it would be helpful to write the code in a way, your future you is able to understand it right away.

So a "good style" would be one, which helps you the most. What that style should look like is an answer I can not give.

I think you do not need OOP or Unit tests for files of 150 LOC. A dedicated VCS would be interesting when you have some evoluting code. Otherwise a .bak does the trick. These tools are a cure to a desease, you might not even have.

Perhaps you should write your code in such a way, that even if you read it while being drunk, you are able to read, understand and modify it.