Best practices build and deploy for on premise applications

Question

I work for a software company which tend to deploy applications on premise for enterprise customers. The software normally consists of server side software (a few microservices) and 1 or many client applications.

The deployed applications are usually a highly customised version of a general application. So we have a separate git repository for each install.

I have been trying to improve the build and deployment cycle however I am struggling to find any best practices for this type of deployment. All the articles I read talk about continuous deployment, but all the related tools seem to be tailored to deploying to the cloud or our own servers.

At the moment we use team city as our build server. At some point we reach a point where the software will be released, so we force a build on the master branch and tag it accordingly.

The binaries are then copied to a dev or production server which the customer owns. And the configuration is done manually by us. Manually mainly means running scripts to setup config files, install services and setup databases.

The problem we are facing at the build and deploy process is very manual. Which means some times there are errors (forgot to run script, typos, missed copying some files)

Is there anything we can do to improve it? Bearing in mind we normally only have access to customers servers through some sort of Van connection and would not be able to deploy to their site without have change requests signed off etc.

Answer 1

To expand on Robert's answer,

Automation is definitely the correct approach. The following should provide a good starting point:

• Ansible: this should be used to actually provision the server. If the Ansible tasks are written correctly, they can be idempotent (run multiple times with no worries), and will only update/change whatever needs to be changed. The typical way to use Ansible is to run it "remotely" using SSH, but I recommend using Ansible in "local" mode (explained below).

• Shell scripts: if anything specific/complex needs to occur on boot, or at the end of the Ansible run, it's best to move it to a shell script which can be called separately. When writing shell scripts, ensure they are POSIX compliant and don't contain too much crazy logic, this way they can run safely on various operating systems, and won't be too difficult to debug down the line.

• Pull, not push: since your customers need to download binaries, you can host them on a local/private HTTPS server, and have your Ansible scripts "pull" (download) the binaries. You can even add an sha256 hash in your Ansible YAML config files to verify the integrity of the downloads. Of course development will be a bit slower if you always need to update the hash in the config files, but if you don't change the binaries too often (ex: once a month), it's an acceptable approach.

• Virtual machines: if you setup a virtual machine, you can test your Ansible and shell scripts on there without affecting your customer's servers. This allows you to catch errors and avoid destroying a production server.

My approach is the following:

• Ensure the server has Git and Ansible installed.
• Ensure the server has an SSH private key to git pull from either GitHub or a private git+ssh server.
• SSH into the server, and perform a git pull, then ansible-playbook -c local

Ansible in "local" mode requires your server to have Ansible installed, but it has the advantage that provisioning is much faster, and doesn't require a network connection (except for the initial git pull).

This method allows your customers to manually decide when a server gets updated, and allows you/them to verify code pushed to the Git repo, before running Ansible on the server. If you're using GitHub to keep track of your Ansible scripts, then you can create a bot account for pulling the repo changes (allowed as per GitHub's ToS), or use your server's public SSH key as a deploy key.

I've listed a few other best practices for building software for on-premises, which you can read about here for additional tips.

Answer 2

Is there anything we can do to improve it?

Automate.

Presumably, your build and deploy process is more or less the same each time. Write some scripts that automate this process. Unlike humans, computers are very good at repeating a mundane process without error.