I am building a system that consists of multiple programs on many machines, some cloud services (such as RDS) and so on.

In an ideal world, I would like to supply some configuration (e.g. deployment keys, AWS credentials) and run a single "deploy all" command that will build and deploy everything.

I would also like it to be smart enough to not rebuild artefacts that have already been built or redeploy infrastructure that already exists.

Currently, I am using Bazel to build my artefacts (.so, .jar, Docker images, etc) and Terraform to provision my architecture (ECs, RDS, etc.).

Each of these tools is very good at what it does, and together, they cover builds and deployments. However, neither does everything (the desired "deploy all" command) and there are cases where they must interact in awkward ways.

For example, suppose I have a microservice written in JavaScript. This is compiled / bundled by Bazel. The bundle is then included in a Docker image along with some secrets generated by Terraform. The Docker image is built by Bazel. Finally, The Docker image is deployed using Terraform!

  1. Bazel builds the application code
  2. Terraform generates / fetches secrets
  3. Bazel builds a Docker image
  4. Terraform deploys the Docker image

I am jumping between the two tools and it doesn't feel like the right way to approach this.

  • Should I wrap Terraform in Bazel and only interact with Bazel?
  • Should I wrap Bazel in Terraform and only interact with Terraform?
  • Should I use some third tool to manage them?
  • How can I resolve this?
  • Can you write a two-line shell script? This doesn't need a "tool". – jonrsharpe Nov 12 '19 at 15:30
  • @jonrsharpe Yes, but that does not scale very well (same reasons I am using Bazel) – sdgfsdh Nov 12 '19 at 15:34
  • Are you using a config server of some sort to resolve those configuration settings? – Berin Loritsch Nov 12 '19 at 17:26
  • @BerinLoritsch Currently not, but if this were required to solve the problem I am open to doing so – sdgfsdh Nov 12 '19 at 17:30

The problem I see is that you are trying to resolve configuration in your build stage, and that is too early. Configuration is at earliest a deploy time concern, and at latest a runtime concern. The challenge is to untie the concepts, and to either leverage existing infrastructure or introduce some to resolve the conflict.

General recommendation:

  • BUILD: Bazel creates docker containers, and defines locations for the secrets/config files
  • DEPLOY: Terraform deploys docker containers, and injects secrets/config files.... OR
  • CONFIG: Use a configuration server to resolve configuration at startup/runtime

Docker containers are not completely closed once they are built. You can inject environment variables, files, or mount volumes when you deploy the container. That is functionality built in to the container technology.

There are several ways to deal with runtime configurations, each with their trade-offs:

  • Spring Cloud Configuration Server: integrates natively with Spring Boot based microservices. Resolves configuration at runtime, and multiple environments are resolved by defining an environment variable to specify the environment for your container
  • Hashicorp Consul and Vault: resolve secrets and handle service discovery, may integrate well with Terraform
  • Kubernetes: Uses ConfigMaps and Secrets, injecting config files into containers as they are spun up.

That's only scratching the surface. Config servers at least allow you to defer the content of certain values until your application is actually spun up for the environment. It's also likely that your current infrastructure can handle runtime configuration, and you simply need to learn how to make use of it.

I believe that once you start thinking of configuration as a separate step, you can clean up your build and deploy stages in your CI/CD pipeline.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.