Python build system

Question

Currently our Python application is deployed like this:

• Development team work on issues, commit code and create a Pull request
• Pull request is integrated to Development branch
• QA and Operations team test & deploy Development branch and then move it to Staging
• If all is good after a week we move it to Production.

Python code is stored in a Git repo and in production servers is just updated (git pull) based on latest Branch (Master), services restarted and all is good.

Today I came across a Google based build system (Bazel). Our python app uses:

• Flask + Celery (external Python libraries: vnc, vmware, ssl, json, logging, math, etc.)

Externally:

• RabbitMQ
• PostgreSQL
• Ngnix
• Lighttpd

For testing we use Unittest.

What would be the benefits of deploying a Build system as part of our Dev pipeline?

Answer 1

Instead of thinking about the potential benefits of a potential tool, think about the actual issues and bottlenecks you encounter in your workflow—only then you should focus on the tools which actually solve those issues.

For instance, you may have a slow feedback cycle between developers and QA. Imagine a developer commits the changes on Monday at 5 p.m.; the working day ends at 6 p.m. The commit contains a regression. Would the developer learn about this regression a few minutes later, and be able to work on a fix when the code is still fresh in her head? Or maybe tomorrow morning, when she arrives at work and still remembers, although not very well, what she did yesterday? Or maybe only on Friday, when she has absolutely no idea what she was doing the day the regression appeared?

By running unit tests, build platforms make it possible to shorten the cycle between the time the regression is introduced and the time the developers are alerted about it. The sorter the better. If it's a few seconds after the “bad” code is introduced, great (this is the benefit of background compilation in some IDEs). If it's a matter of minutes, it can work. If it's a week, chances are you're wasting hours of painful debugging.

Or maybe you want to know the health of the project: an increase in failed builds usually indicates a problem. A build system usually makes it possible to visually identify such trends over days or months or the entire life of the project.

Or you may notice that developers spend time working on adding features, while there are blocking issues. They commit non-working code, their colleagues check out this code and are unable to progress or work effectively. Suddenly, most members of the team are found with code which won't compile, and all try to solve at the same time the same issue, leading to wasted hours and a mess.

A build platform would help here, by providing a visual of build failures. If a build fails, this should be an immediate response of a team: other members should avoid updating to the problematic revision; the author of the commit which caused the build to fail should focus on fixing the build (eventually soliciting help from pairs).

Maybe you simply want to give QA and Ops people a possibility to get back to an older revision either to test if a bug already existed or because the more recent revision introduces a blocking regression in production. In this case, a build system coupled with artifact repository will let them switch with ease between different revisions, without having to contact the developers.

Those are four cases of eventual issues which could be solved by a build platform. Look at the actual issues you encounter and:

• Either search for the tool or a change in your workflow which solves the issue,

• Or describe those issues on Programmers.SE, asking what should be done to solve them.

It might be that you actually need a build server, or a Continuous Integration platform. Or maybe you need to introduce DevOps in your organization. Or a simple tiny change in the way things are done is largely enough. Remember to focus on your issues, independently of the actual tool:

“If all you have is a hammer, everything looks like a nail.”

Answer 2

What a build system like Bazel might give you?

• Self-contained builds with all dependencies, including all binary extensions dependencies, put into one package (see py_binary). This additionally helps keep as few things installed on a prod machine as possible. limiting the attack surface.
• Easily invoked unit tests in different environments.
• Compiling and packaging anything else, like your JS, CSS, or custom binaries, along the way.

It does take time and effort to set up. It pays with larger projects; it might be an overkill for smaller projects. (I used Bazel at Google where the infrastructure was already set up for me; I never tried to set up Bazel myself but it does not look like a complete piece of cake.)

There is a number of smaller build systems, with different goals and approaches. Python apps are not that hard to deploy; your favorite OS's package manager might suffice (we built plain Debian packages at another large internet company; today I'd consider guix, too).

What I'd like you to think about is rather:

• Do you have reproducible builds, and what is the easiest way to achieve them? (Bazel, among others, is fine for that, but not necessary.)
• Can you do reproducible, click-a-button deployments? (Bazel helps achieving that, but not necessarily in the fastest or easiest way possible.)
• Are your builds fast? (Bazel does a reasonable job of incremental builds.)
• Are your deployments fast? (Bazel neither helps nor precludes that.)
• Is your dev / test / QA environment sufficiently similar to the prod environment?

After you have answered these questions (maybe you have already), you'll see if Bazel, or any other build tool, is a good fit for your workflow, and / or what qualities of a build tool are desirable for you.