Blurred lines between deployment (Terraform) and build processes (Bazel) leading to an awkward build and release process

Question

I am building a system that consists of multiple programs on many machines, some cloud services (such as RDS) and so on.

In an ideal world, I would like to supply some configuration (e.g. deployment keys, AWS credentials) and run a single "deploy all" command that will build and deploy everything.

I would also like it to be smart enough to not rebuild artefacts that have already been built or redeploy infrastructure that already exists.

Currently, I am using Bazel to build my artefacts (.so, .jar, Docker images, etc) and Terraform to provision my architecture (ECs, RDS, etc.).

Each of these tools is very good at what it does, and together, they cover builds and deployments. However, neither does everything (the desired "deploy all" command) and there are cases where they must interact in awkward ways.

For example, suppose I have a microservice written in JavaScript. This is compiled / bundled by Bazel. The bundle is then included in a Docker image along with some secrets generated by Terraform. The Docker image is built by Bazel. Finally, The Docker image is deployed using Terraform!

1. Bazel builds the application code
2. Terraform generates / fetches secrets
3. Bazel builds a Docker image
4. Terraform deploys the Docker image

I am jumping between the two tools and it doesn't feel like the right way to approach this.

• Should I wrap Terraform in Bazel and only interact with Bazel?
• Should I wrap Bazel in Terraform and only interact with Terraform?
• Should I use some third tool to manage them?
• How can I resolve this?

Answer 1

The problem I see is that you are trying to resolve configuration in your build stage, and that is too early. Configuration is at earliest a deploy time concern, and at latest a runtime concern. The challenge is to untie the concepts, and to either leverage existing infrastructure or introduce some to resolve the conflict.

General recommendation:

• BUILD: Bazel creates docker containers, and defines locations for the secrets/config files
• DEPLOY: Terraform deploys docker containers, and injects secrets/config files.... OR
• CONFIG: Use a configuration server to resolve configuration at startup/runtime

Docker containers are not completely closed once they are built. You can inject environment variables, files, or mount volumes when you deploy the container. That is functionality built in to the container technology.

There are several ways to deal with runtime configurations, each with their trade-offs:

• Spring Cloud Configuration Server: integrates natively with Spring Boot based microservices. Resolves configuration at runtime, and multiple environments are resolved by defining an environment variable to specify the environment for your container
• Hashicorp Consul and Vault: resolve secrets and handle service discovery, may integrate well with Terraform
• Kubernetes: Uses ConfigMaps and Secrets, injecting config files into containers as they are spun up.

That's only scratching the surface. Config servers at least allow you to defer the content of certain values until your application is actually spun up for the environment. It's also likely that your current infrastructure can handle runtime configuration, and you simply need to learn how to make use of it.

I believe that once you start thinking of configuration as a separate step, you can clean up your build and deploy stages in your CI/CD pipeline.

Answer 2

The problem here is that you're hard-coding secrets into your containers. It's better to externalize the secrets, and then wire them in at deployment time.

This has several advantages:

1. Your image becomes portable between environments. You can test it in one environment (say dev or stage) and once you've verified it, you can then deploy it to the next environment. (Or you can pull your production image into your development environment and test the real image, not a replica that you hope was built identically)

2. If your production image becomes public, then the public doesn't have your production secrets.

3. Since your images don't change between environments you can do things like signing verified images, and archiving your images.

4. Credential changes don't require rebuilds, allowing much more operational freedom. This is a big thing in a security incident.

5. You can use different policies to guard your code and your secrets.

These are a few of the advantages to getting the secrets out of your containers, but the list goes on.

Hard-coding your secrets into your container isn't really any different than putting your secrets into your application code. Containers are just a bigger application binary that carries along much of the environment, and all the traditional advice against hard-coding secrets applies to them as much as it does to applications.

Once you get the secrets out of the containers, the rest of your build environment becomes much simpler to deal with too.