3

I can't for the life of me find any reasonable guides or recommendations on how to properly install or deploy applications made of multiple processes. Some considerations:

  • Multiple machines
  • Multiple services
  • Data and data migration
  • Different environments (eg dev vs QA vs production)
  • Configuration
  • Secret keys
  • handling and minimizing downtime
  • logs
  • Automated vs manual installation steps

For example, let's say you have an application where you want 3 web servers on 3 different machines, a primary database on another separate machine, and some miscellaneous services on a 5th machine (eg logs db, service monitor, etc). Let's say further that you're releasing this service as a package that other people can use, so that many users of this package have various different versions installed. Let's say even further that in version 4, you update your mongo database from Mongo 2.6 to Mongo 3.2 (which requires multiple steps of data migration).

What are the best practice techniques for dealing with this kind of thing in an automated way, minimizing the points where the service administrator has to perform manual steps?

I could imagine a situation where you have an overall cluster configuration, which defines the configuration for each machine, and each machine-configuration in turn defines the configurations for the various services to be installed on that machine. Sort of like NixOS and friends. But how do you define data migration steps that may have dependencies on the state of other services in the system?

I've tried to think about this on my own and have done various research, but I've only managed to find super basic descriptions of service oriented architecture and inane advice like "use a batch file" or "use this worthless installer library like gulp" or "this is how we use git in our installation process", but nothing about the handling of data migration, especially in the context of a service oriented architecture. Maybe I'm searching for the wrong things? Anyone have a lead on this kind of thing?

Improve this question
9
  • 8
    Your question is interesting but I feel it may be too broad in scope. The answer you are looking for will likely involve use of standard packaging for your environment (DEB, RPM, etc.), containerization and related services (Docker, swarm, etcd, etc.) as well as service management and deployment strategies (infrastructure as code tooling, immutable server, etc.). For anyone to write a proper answer that covers most of your question will take quite some time. ^_^;;;
    – KevinLH
    Commented Mar 2, 2017 at 10:51
  • 1
    How does the description you provided of installation problems differ from that of any other software (that is not "service-oriented design")?
    – Robert Harvey
    Commented Mar 2, 2017 at 14:32
  • 1
    Why is asking a question on “best practice” a bad thing?
    – user22815
    Commented Mar 3, 2017 at 23:05
  • 1
    @BT "best practice" is synonymous with "I am a cargo cult programmer, tell me what everyone else does so I can use it." Same as with questions asking "what design pattern should I use here?" Instead, focus on what the correct solution is regardless of what is the popular flavor of the month technology or method. In the end it is more a matter of phrasing the question than anything, but it goes a long way toward demonstrating one's ability to solve problems, rather than follow the crowd for better or worse.
    – user22815
    Commented Mar 4, 2017 at 19:14
  • 1
    @BT These kinds of problems are relatively new. If you look how the industry shiftet even in the last 7 years. There isn't consolidated knowledge about how to do things. There are only scars and ways to circumvent some kinds of scars.
    – Thomas Junk
    Commented Oct 2, 2017 at 14:43

2 Answers 2

Reset to default
5

It is hard to give any concrete advice, since the only things which could be said are far too general to help you in your concrete problem domain.

What you are mostly looking for is some kind of automation with tools like:

From which I started looking into Ansible lately. Its selling point which makes it attractive for our company is, that it is agentless and uses mainly ssh.

For storing secrets, there are several options:

What all these tools have in common is a declarative approach to system states. You declare the desired outcome of your system. Which is by the way the driving force behind systemd (but that is another topic).

There are differnt kinds of deployment scenarios:

In case you live in Dockerland:

What are the best practice techniques for dealing with this kind of thing in an automated way, minimizing the points where the service administrator has to perform manual steps?

As far as this part of the question goes, the tools above could give you inspiration to minimize manual intervention. As you see in the range and variety of tools, there is no one size fits all solution.

What these tools could do: drive the way, you built

What these tools couldn't do: plan the way

This part is up on you: careful planning

There are also strategies, which help you along the way like Blue Green deployment, evolutionary DB design etc.

But how do you define data migration steps that may have dependencies on the state of other services in the system?

That depends on the concrete scenario. There is no recipe to follow.

Mostly divide et impera / divide and conquer is the rule to follow: do things incrementally step by step with always having a PlanB (or even C and D).

Maybe I'm searching for the wrong things?

No. You aren't. All you would find is basic introductory material, since the interesting parts aren't doable with a simple HOWTO.

Anyone have a lead on this kind of thing?

The problem with that is: When you ask, say Google, Amazon, Facebook, Netflix, etsy etc. you will hear about their problems and their solutions, but that won't help you in any way.

Reading tips:

Improve this answer
1
  • 1
    Thanks for the answer! It looks pretty in depth. I'll have to go over all this when I have more time
    – B T
    Commented Oct 5, 2017 at 20:33
0

So this isn't a full answer, but I can mention where I'm at in terms of my current design with the project I'm working on. I have a single repository (which should be split out at some point) that contains code for a couple services, including a main application, a forwarder that facilitates A/B deployment, a backup script, and an application watcher. I also need a mongo database. The application watcher is the only thing run on a separate server.

So when I deploy on my development machine, I use npm install which pulls in all my npm dependencies, then runs a postinstall.js script, which runs a set of installation steps. The way these installation steps work is that there's an ordered list of installation steps defined in some file that does things like initializes configuration override files, generates passwords / tls keys / etc, installs the mongo database, creates indexes, etc. To keep track how many steps in that postinstall scripts have been completed, there's a version number stored in a file outside of version control. This allows postinstall.js to also be in charge of data migration. If I need to migrate data, I can make it a step in postinstall.js. I also have a separate script that watches files and auto-builds frontend bundles (an app bundle and a config bundle) on change of relevant files.

Production works somewhat similarly. For production, I package up the code into an archive I send over to the server and execute, which installs a new version of the code over the inactive folder (the folder the forwarder I mentioned is serving from its inactive port, as opposed to the active port our customers are supposed to be suing). After installation, I can switch which folder is active so that customers will see the new build. Doing this with no downtime requires that data-migrations are done in 2 steps:

  1. Update the code to store data in a format that old code will still recognize in addition to the planned format the next version of the code will need that data in.
  2. Run a data-migration script to update any old data to the new format and update the code to stop storing data in the old format and start using the new format.

As for configuration, I wrote about how we do that here: https://stackoverflow.com/questions/5869216/how-to-store-node-js-deployment-settings-configuration-files/42545491#42545491 . Any secrets (keys etc) are either generated on-machine or are manually put into a keys file on the machine.

But this method doesn't have any definition of dependency between the services, and most of those services are tightly coupled. I often get bogged down in how to properly define the installation steps such that they'll work for updating production from its current version and work for installing the system from scratch.

Update:

Some points of techniques for each item:

  • Multiple machines
  • Multiple services
    • Docker
  • Data migration
    • Exhaustive data sanity checks (before and after a migration) (kinda like unit tests for your data)
    • Data cleansing when before-migration sanity checks fail
    • seamless migration: Versioning individual data objects, then after the application loads the data object, converting (up or down) from the canonical version to the version required by the current application version, and then back (down or up) to the canonical version before save. Eliminates the need for lengthy migration scripts and allows for different application versions that requiring different data versions. If the canonical version is the version used most by applications using the data (which would be optimal), you need a separate service that older application versions are pointed to which would convert from and to newer data versions (since those old versions wouldn't know about the newer format). If the canonical version is the version used by the oldest active version of the application (less optimal), this isn't needed.
    • 4 stage migration: update the database to support both new and old versions, update the application to support both new and old versions, migrate all old versions, then lastly remove application and database support for old version
    • 3 stage migration: shutdown application servers, migrate data, then and redeploy the new version of the application
  • Different environments (eg dev vs QA vs production)
    • Use configuration files that describe all differences between environments
  • Secret keys
    • Secret keys should never be version controlled with either the application code nor the configuration files (so exposure of keys is minimized)
    • Secret keys should only be exposed to the machines that use the keys, and any key backup machine or key distribution machine.
  • handling and minimizing downtime
    • Blue/green deployment (aka a/b server deployment or active/inactive deployment)
    • Service-monitoring services that generate alerts when services become unexpectedly unavailable. Instances of these service watchers should be deployed on at least 2 separate machines so that one watcher can watch the other (otherwise when the watcher goes down, nobody knows).
    • Service-restarting services (eg systemd)
  • logs
  • Automated vs manual installation steps
    • Package managers (npm, pip, yum, rpm, etc)
    • Machine configuration managers (NixOS et al, Ansible, Puppet, etc)

Some additional sources of information on each item:

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.