1

I work for a software company which tend to deploy applications on premise for enterprise customers. The software normally consists of server side software (a few microservices) and 1 or many client applications.

The deployed applications are usually a highly customised version of a general application. So we have a separate git repository for each install.

I have been trying to improve the build and deployment cycle however I am struggling to find any best practices for this type of deployment. All the articles I read talk about continuous deployment, but all the related tools seem to be tailored to deploying to the cloud or our own servers.

At the moment we use team city as our build server. At some point we reach a point where the software will be released, so we force a build on the master branch and tag it accordingly.

The binaries are then copied to a dev or production server which the customer owns. And the configuration is done manually by us. Manually mainly means running scripts to setup config files, install services and setup databases.

The problem we are facing at the build and deploy process is very manual. Which means some times there are errors (forgot to run script, typos, missed copying some files)

Is there anything we can do to improve it? Bearing in mind we normally only have access to customers servers through some sort of Van connection and would not be able to deploy to their site without have change requests signed off etc.

0

Is there anything we can do to improve it?

Automate.

Presumably, your build and deploy process is more or less the same each time. Write some scripts that automate this process. Unlike humans, computers are very good at repeating a mundane process without error.

  • At the moment we write separate scripts which install services, setup databases and setup config files. I think from some of the blogs I was reading I was hoping for a process which was: accept a pull request and have a single script created which would could install or update an entire environment. Thinking about it there is no reason we can't. Are there any tools which help with this, or would it be best to wrote the scripts by hand, as part of the committed code? – Nick Williams Sep 9 '16 at 22:38
  • 1
    We use Jenkins. From Jenkins we have access to envs vía scp, we can also run remote scripts, etc. If Jenkins taskis well parametrized you could trigger the build and do deploy of any of your Git branches. Jenkins also have API web, so you can implement a client side app to trigger Jenkins tasks – Laiv Sep 10 '16 at 8:05
3

To expand on Robert's answer,

Automation is definitely the correct approach. The following should provide a good starting point:

  • Ansible: this should be used to actually provision the server. If the Ansible tasks are written correctly, they can be idempotent (run multiple times with no worries), and will only update/change whatever needs to be changed. The typical way to use Ansible is to run it "remotely" using SSH, but I recommend using Ansible in "local" mode (explained below).

  • Shell scripts: if anything specific/complex needs to occur on boot, or at the end of the Ansible run, it's best to move it to a shell script which can be called separately. When writing shell scripts, ensure they are POSIX compliant and don't contain too much crazy logic, this way they can run safely on various operating systems, and won't be too difficult to debug down the line.

  • Pull, not push: since your customers need to download binaries, you can host them on a local/private HTTPS server, and have your Ansible scripts "pull" (download) the binaries. You can even add an sha256 hash in your Ansible YAML config files to verify the integrity of the downloads. Of course development will be a bit slower if you always need to update the hash in the config files, but if you don't change the binaries too often (ex: once a month), it's an acceptable approach.

  • Virtual machines: if you setup a virtual machine, you can test your Ansible and shell scripts on there without affecting your customer's servers. This allows you to catch errors and avoid destroying a production server.

My approach is the following:

  • Ensure the server has Git and Ansible installed.
  • Ensure the server has an SSH private key to git pull from either GitHub or a private git+ssh server.
  • SSH into the server, and perform a git pull, then ansible-playbook -c local

Ansible in "local" mode requires your server to have Ansible installed, but it has the advantage that provisioning is much faster, and doesn't require a network connection (except for the initial git pull).

This method allows your customers to manually decide when a server gets updated, and allows you/them to verify code pushed to the Git repo, before running Ansible on the server. If you're using GitHub to keep track of your Ansible scripts, then you can create a bot account for pulling the repo changes (allowed as per GitHub's ToS), or use your server's public SSH key as a deploy key.

I've listed a few other best practices for building software for on-premises, which you can read about here for additional tips.

  • 1
    In my opinion this should be the accepted answer, it provides more useful direction. – jQwierdy Aug 3 '17 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.