I can't for the life of me find any reasonable guides or recommendations on how to properly install or deploy applications made of multiple processes. Some considerations:
- Multiple machines
- Multiple services
- Data and data migration
- Different environments (eg dev vs QA vs production)
- Configuration
- Secret keys
- handling and minimizing downtime
- logs
- Automated vs manual installation steps
For example, let's say you have an application where you want 3 web servers on 3 different machines, a primary database on another separate machine, and some miscellaneous services on a 5th machine (eg logs db, service monitor, etc). Let's say further that you're releasing this service as a package that other people can use, so that many users of this package have various different versions installed. Let's say even further that in version 4, you update your mongo database from Mongo 2.6 to Mongo 3.2 (which requires multiple steps of data migration).
What are the best practice techniques for dealing with this kind of thing in an automated way, minimizing the points where the service administrator has to perform manual steps?
I could imagine a situation where you have an over cluster configuration, which defines the configuration for each machine, and each machine-configuration in turn defines the configurations for the various services to be installed on that machine. Sort of like NixOS and friends. But how do you define data migration steps that may have dependencies on the state of other services in the system?
I've tried to think about this on my own and have done various research, but I've only managed to find super basic descriptions of service oriented architecture and inane advice like "use a batch file" or "use this worthless installer library like gulp" or "this is how we use git in our installation process", but nothing about the handling of data migration, especially in the context of a service oriented architecture. Maybe I'm searching for the wrong things? Anyone have a lead on this kkind of thing?