Clarifying the steps in a CI/CD, but namely if if unit testing should be done building a Docker image or before

Question

I'm building at a Build and Deployment pipeline and looking for clarification on a couple points. In addition, I'm trying to implement Trunk Based Development with short-lived branches.

The process I have thus far:

1. Local development is done on the main branch.

2. Developer, before pushing to remote, rebases on remote main branch.

3. Developer pushes to remote short-lived branch: git push origin main:short_lived_branch.

4. Developer opens PR to merge short_lived_branch into main.

5. When PR is submitted it triggers the PR pipeline that has the following stages:

   1. Builds the microservice.
   2. Unit tests the microservice.
   3. If passing, builds the Docker image with a test-latest tag and push to container registry.
   4. Integration testing with other microservices (still need to figure this out).
   5. Cross-browser testing (still need to figure this out).

6. If the PR pipeline is successful, the PR is approved, commits are squashed, short_lived_branch is merged to main and short_lived_branch is deleted.

7. The merge to main triggers the Deployment pipeline, which has the following stages:

   1. Builds the microservice.
   2. Unit tests the microservice.
   3. If passing, builds the Docker image with a release-<version> tag and push to container registry.
   4. Integration testing with other microservices (still need to figure this out).
   5. Cross-browser testing (still need to figure this out).
   6. If passing, deploy the images to Kubernetes cluster.

I still have a ton of research to do on the integration and cross-browser testing, as it isn't quite clear to me how to implement it.

That being said, my questions thus far really have to do with the process overall, unit testing and building the Docker image:

1. Does this flow make sense or should it be changed in anyway?

2. Regarding unit testing and building the Docker image, I've read some articles that suggest doing the unit testing during the building of the Docker image in the Dockerfile. Basically eliminating the first two stages in my PR and Deployment pipelines. Some reasons given:

   • You are testing the code and not the containerized code which is actually what will be run.
   • Even if unit testing passes, the image could be broke and it will be even longer before you find out.
   • Building on that, it increases the overall build and deployment time. From my experience, the first two stages in my pipelines for a specific service take about a minute and half. Then building and pushing the image takes another two and half minutes. Overall about four minutes. If the unit tests were incorporated into the Docker build, then it could possibly shave a minute or more off the first three stages in my pipeline.

   Would this be a bad practice to eliminate the code build and unit testing stages, and just moving unit testing into the Docker build stage?

Thanks for weighing in on this while I'm sorting it out.

Answer 1

Both

Unit Tests

Unit test are test for a small piece of code within a larger binary. These tests shouldn't care about the docker image or any other issue.

Though as its intended to run within docker, within a given base image, it does stand to reason that you should run these tests within that platform to eliminate platform variability. But if there is platform variability its not your unit tests that you need to be looking at, but the platform itself.

Integration Tests

Integration tests are aimed at larger systems of code for example a deployable service. These tests require the environment to reflect the real one as much as possible, and as such they would apply to testing you code within its assembled docker image.

---

As such you would:

• Checkout, and gather any other dependencies
• Build the binary executables and libraries from source
• Run unit tests to verify that the source is well-formed
• Build the docker image from the binaries just built
• Deploy the docker image
• Run the integration Tests against the running docker image, using as life-like an environment as possible
• Push the docker image into a repository for formal deployments.

Answer 2

The purpose of the container is to remove variance in the environment. Unit tests shouldn't have any dependence on the environment. You would expect them to run the same either way.

You have integration tests, which should catch any errors anyway. So really its going to come down to practical considerations.

• Do you want to push failed images to the registry?
• How long does it take to build the image?

etc etc

Answer 3

Never, ever build in docker images, because it means we have to install dev-kits in the container. No docker container in production should have any of these kits. For security. ¹

On the other hand, building the app should happen in an environment already set for the purpose.

Once, I made a docker image based on Debian that, years later, reached its EOL. In consequence, some of the package repositories were closed and no longer available. The image had to download and install an O.S package, from a package repository that was no longer available due to the end of the LTS. Now imagine happening to several docker images, at the same time in different stages of the CI. It's a big stopper in your CI.

Build the Docker image only once, push it to the registry only once, deploy it anywhere and anytime.

Finally, I agreed with @Kain0_0 with

Unit tests are tests for a small piece of code within a larger binary. These tests shouldn't care about the docker image or any other issue.

Unit tests should be independent of the environment. These tests should not depend on anything but production code, which is bundled in the same project. Bear always in mind this

We should be able to run unit tests anytime, anywhere in any order and always get the same results.²

Integration tests, load tests, smoke and monkey tests and test of the like E2E, indeed are better performed on applications running in environments similar (if not exactly equals) to the production. Of course, it does mean deploying changes first in a sort of stage environment before the Deployment pipeline deploys change to production.

---

^{1: Even if you think that you can remove these libs after the build, such a thing will generate one or more layers in the docker image that potentially will make it grow in size! It might interest.}

^{2: We don't run unit tests locally by building a docker image first and running a container later... Countless times per day, any time of the day. It's inefficient and a waste of time.}