We are developing some Python applications (mostly CLI) for use only internally in the company. Hundreds of machines on which the applications may run. Instead of installing the apps on each machine, we would like to use a shared network filesystems that they can all access as the deployment target. Some packages need to be shared between the apps of course.

How do we do this? Seems setuptools, pip and virtualenv are more oriented towards per-machine installations. I want my release to make the tools immediately available to any user on any machine. Note no root access.

Any recipes are welcome.

  • 1
    Are you using software like Puppet or Ansible to manage your workstations, or are they configured "by hand" one at a time? – Kevin Nov 26 '18 at 20:04
  • No configuration... Just put the latest in a folder and change a 'current' symlink to point to it. Next run on any machine will pick up the release. – Dave Nov 30 '18 at 20:36

Creating a virtualenv on a shared filesystem should work, though end users would still have to activate it themselves. Of course, all a virtualenv really does is to symlink some executables to sit at the front of your PATH and to make sure the virtualenv is in your PYTHONPATH – you may be able to write scripts that do this manually.

There are a couple of gotchas you should be aware of before choosing this deployment method:

  • Shared file systems are often comparatively slow. This will make any import more costly.
  • Anything that may be executable must not be writable by other users, or that could be abused to engineer a wormable malware against your organization. Defense in depth is important.
  • CPython caches the compiled bytecode in *.pyc files. As the directory must not be writable, these files should be created during installation. But these files are only useful if everyone uses the same Python version. If you can deploy a specific Python version, you could also just deploy your software directly.
  • If your end user systems have different operating systems or CPU architectures, dealing with native libraries is more difficult. You can no longer offer a single virtualenv, but must offer one per OS/architecture combination. This also makes it more difficult for end users to set up their system correctly.

Since you still have to at least deploy Python and virtualenv setup scripts to end users, it really might be easier to use those deployment mechanisms for your Python software as well.

Also consider whether running the software on end-user systems is appropriate. Web-based software gives you a lot more flexibility about deployment. For example, in the Data Science community Jupyter notebooks are a common tool. You can install Jupyter locally via pip. But an organization may prefer to host them on a beefy server with a guaranteed-working set of dependencies – a win for everyone involved.

  • Thanks for the detailed response. The tools in question are not web-related, just nightly regression automation stuff dropped to all machines that will run the nightly flows. – Dave Nov 30 '18 at 20:41
  • @Dave In that case, it might be a lot simpler if the nightly job doesn't execute the regression software directly, but first executes a script that updates and installs your Python software locally. That's exactly the same deployment experience for you, but doesn't suffer from most of the drawbacks I mentioned. You also might want to consider whether deploying Python scripts is appropriate, or whether e.g. a Docker image could avoid potential problems. – amon Dec 1 '18 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.