Preferred way to expand a command line script to be used as a library in Python

Question

I have a useful Python script that I've been invoking from the command line. It has decent number of options, maybe 20, and it's not unusual to run the script with six or seven flags. Then the rest of the input comes via stdin.

Now I have some other Python code from which I'd like to call this useful little utility. Two options I can think of are:

1. I can use subprocess.call and invoke my little script
2. (A little better) I can cobble together a command line and then pass it as a list of strings to argparse
3. I can totally refactor the program so that the entry point of my utility is Python function call and then have my command line utility just call this function. In principle this seems like the responsible thing to do, but it does leave my managing two separate interfaces to my function. For example, I have to decide whether I want to let argparse know the default values for my options or have the function know the defaults (or have two sets of defaults). Any validation I do using, say, ArgumentParser.add_mutually_exclusive_group will not apply when my tool is run as a library instead of a command line script.

Is there a standard paradigm for creating a single interface in Python that is well-suited to being invoked both from Python and from the command line?

Answer 1

If there is not a good reason to not do so, I would definitely advocate a spin on option 3. As @jonsharp mentions, breaking up your utility into clean units of functionality is a good way to ensure testability. Even the smallest scripts can eventually morph into a much larger program and making sure that you have an extensible API sooner rather than later will alleviate much headache down the road.

The way I'd approach this is:

1. Break up your code into logical methods with clean and clear I/O
2. Add unit tests. Having them is never a bad thing.
3. Rather than using if __name__ == '__main__', create a main() (or similar) method containing your entry point
4. Use setuptool's setup() function to define the script entry point in your setup.py file.

For example:

from setuptools import setup
setup(
    name='mypackage',
    version='0.1',
    entry_points={
        'console_scripts': [ 'myscript = mypackage.mymodule:main' ],
    }
)

Now, not only is all of your code (including main()) is easily unit testable, but you can still have your console entry point once you've done a python setup.py install|develop.

Any validation I do using, say, ArgumentParser.add_mutually_exclusive_group will not apply when my tool is run as a library instead of a command line script.

Depending on how your API is designed, you may need to add some extra validation to input parameters, but that should likely be there to prevent unexpected input anyways.

Edit: The only time I would use generally use subprocess is when I'm calling into a non-Python application or another Python script that I don't own or have the time to refactor, but the latter only being as a last resort. Most well-written Python utilities will expose both command line utilities and internal API.

Answer 2

My guess is that you can pursue #3 by way of #2 and that it won't require anything close to a total refactor. Often in these situations, you just need to make a few adjustments at the entry points and (sometimes) exit points.

• If needed, wrap your current script in a main() function.

  def main(args, stdin = None):
      if stdin is None:
          stdin = sys.stdin
  
      # full script here
  
  if __name__ == '__main__':
      main(sys.argv[1:])
  

• That function should take a list of strings. When you parse command-line options, operate on args, not sys.argv.

• Similarly, adjust other parts of your script to avoid direct operations on sys.stdin and related streams. Instead, use the arguments passed to your main() function.

Programatic users (as opposed to command-line users) will pass in a list of strings and any open file handles (or iterables) they want the code to use. Later, if needed, you can make the programatic use more natural by writing a function that knows how to convert typical positional and keyword-style arguments into the list of strings that your option parser needs.

Answer 3

Calling a Python script A from another Python script B through subprocess looks weird to me. I'm not sure if performance impact will matter, but still, having a dependency on A in B and being able to call public methods directly seems more powerful and convenient, especially if both A and B use pip.

I would probably use subprocess in one of those cases:

• If you don't use pip. Dependency handling and versioning can quickly become too complicated; calling A through command line will then be a better choice by providing a better abstraction.

• If the script A is not written by you or your company and is intended to be used through command line.

  For instance, vmbuilder—the tool which creates virtual machines in some distributions of Linux, is intended to be used from command line, and it seems natural (and easy) to use it this way instead of invoking Python methods directly.

• If there is a risk for script A to be rewritten in another language. By using the command line, it doesn't matter which language A uses under the hood. It can be Java, C or Haskell, or may even be an ordinary Bash script—the way you call it will still be the same.