Best Practices For Temporary Scripts (Python)

Question

I am running two separate programs which are similar enough that they share a lot of code. I run these programs often and after I evaluate the output. This is a very fluid process and everytime is different from the last time.

Currently I write a script named something like: "temp_evaluate_program_A" and then I will copy it to "temp_evaluate_program_B" and change a few lines, along with writing a lot of code on the fly. Often the code isn't very readable and only parts of it will ever be reused. This seems suboptimal as I get a lot of duplication.

What do other people do in these fluid situations where there is scope for sharing code between these related programs and reusing the code over time but the scope is limited.

Answer 1

I have this problem when writing data manipulation scripts. Each does its own thing--but I'd like there to be as much commonality as possible, and I'd like each script to be as simple as possible.

A few suggestions:

1. Modules. Learn to package your reusable code into modules. Even among a collection of single-use, temporary programs there will tend to be common tasks. You'll often need to read and write data from similar sources, for example, and often in common formats. Great! We already have some functions that can be coded into shared modules. In the simplest case, this can be just a file shared.py (or whatever you'd like to call it) in the same directory as your scripts. Then use from shared import * in each of your scripts. As your modules become more sophisticated, you can learn to package them so they can be installed as standard modules on your system, or even shared with others. That is a bit of a learning curve; I recommend starting small.

2. External modules. Gradually learn to use code from other modules. If you're not familiar with the Python Package Index, it's a great source of pre-packaged functionality to do many things. Installing most code there is a simple pip or easy_install command away. Over time you will find a handful of modules from either the Python standard library or PyPI that greatly simplify your work. Learn them and rely on them often. If you're working with Web pages, for example, requests is a wonderful tool. For dates and times, I find arrow very helpful. And I'd (not so) humbly add my own say, show, and textdata modules as other examples of modules that try to not just provide functions, but make otherwise messy tasks simple, clean, and self-contained.

3. Common idioms. Even if you cannot reuse code directly, using common patterns of coding will reduce your cognitive load as you work. You will see the same patterns of usage time and again, becoming a master of those idioms, and quickly comprehending the intent and effect of each idiom. Python is an especially good language for idiomatic development, as has a strong community ethos of using shared design principles (e.g. "the Zen of Python") and common programming styles (e.g. PEP-8).

4. Source Code Control. If you're using a lot of individual scripts, and especially if you're building many of them ad hoc, you would benefit from source code control. You might think for one-off coding, source repositories wouldn't have that much value, but I've found quite the opposite. The more fluid your programming situation, the more having some elements that provide discipline and structure becomes. It lets you make mistakes, screw up your code, then find out "what did I change?" And if you like, revert to what you had previously. I like Mercurial. It's easy to get going, and excellent tutorials are available. Git is also popular and effective. Commit early; commit often.

If your individual scripts are idiosyncratic and do quite different things, then your opportunities for pure reuse are, if not minimal, certainly constrained. That doesn't mean there aren't opportunity for leverage. It turns out that leverage--the ability to get a disproportionate reward for the effort expended--is the real goal. Reusing your own code is one way of getting it. Using others' code and expertise (external modules), using high-productivity idioms, and providing some discipline and order in what is otherwise a messy situation--those are other useful mechanisms to get more out of each working hour.