0

Our CI process goes as this cycle (I think it is quite normal), unit test => build docker image => run function test against the image => if test fails remove the failed docker image, figure out the reason and build a new docker image.

But checking the test result and removing the failed images has become a labor-intensive job so I have been thinking how to improve our CI process. I now design a process that if the container fails the smoke test (or even worst, fails to start), I will make docker build fails, so no more failed docker image.

But I face some technical difficulties, e.g. for my node.js project, if I just add RUN npm run start in Dockerfile, not as the entrypoint but in order to test it, and if it fails to run, docker build indeed fails. But if npm run start succeeds, docker build won't exit then. I have to design some kludge/clever way to bypass this.

I do some research about my "improved" CI process but I can't find others do that. So is there problem with that ? My goal is to reduce failed docker images. If this process does have problem(s) what other options can I reduce failed docker images?

We have dedicated docker build servers, so the docker build time is not a concern here and as why removing the failed images has become a labor-intensive job it may deserve another question.

PS. I searched the similar questions here I can only find these 3, they are not my question.

  1. In CI, should tests be run against src or dist?
  2. Clarifying the steps in a CI/CD, but namely if if unit testing should be done building a Docker image or before
  3. Should I include tests in Docker image?
2
  • If your goal is to reduce failed Docker images: what's leading to failures? Taking known-good code and building a container from it should be a fairly safe operation.
    – jonrsharpe
    Mar 24, 2022 at 8:07
  • It is like a remedy. Yes I agree the better way to "taking known-good code and building a container" but the reality is as a micro service app, it has gone wrong for several reasons in the past, so testing it using docker to remove variance in the environment is the only option we have. Then the next question is to reduce failed images. Mar 24, 2022 at 8:28

3 Answers 3

2

The Dockerfile isn't the right place to run tests. It should fail when the container can't be built for some reason, not when the software it contains somehow behaves incorrectly.

Instead, you should automate testing the created image and make pushing the image into your organization's docker registry dependent on a successful test. On unsuccessful tests, you can automatically remove the image from the build machine's local storage.

So your CI script should do something like this:

  • Build container image.
  • If unsuccessful, abort as failed.
  • Run test in the container image.
  • If unsuccessful, remove image and abort as failed.
  • Else push image to your registry and possibly remove it locally to avoid clutter.

Send notifications about failure/success as you see fit.

5
  • Thanks for answering my question. What you said is what we did before, except our CI script always pushes to our local registry after build, so it makes removing the failed image labor-intensive. But I want to focus on what is the problem with my new process. I know many people will say it is not the the right place to run tests or CI should go in build, test, release cycle. But what exactly is the problem? Mar 24, 2022 at 8:49
  • 1
    One major aspect is that a docker image should be as pristine as possible. If you already run tests in it, there are probably test artefacts and other residues left in the finished image. Of course it may be possible to somehow run a test script that starts up npm in background, does some checks, and kills the npm process, but that feels rather kludgy. Fixing the CI pipeline to test images and push only good ones is just cleaner. So in my opinion, the problem is that your proposed process is dirty - you might want to live with that if no other solution is viable. Mar 24, 2022 at 9:30
  • Yes I agree it is dirty. Give me some time to think it through before I mark your answer accepted. Thanks. Mar 24, 2022 at 9:52
  • No problem, you should probably wait for other answers with a different perspective before accepting one. Mar 24, 2022 at 11:47
  • Hi I added my own answer after some test and would like to see your comment. Mar 30, 2022 at 7:47
2

I think your problem stems from the docker philosophy of compiling your code in a docker file.

This is a popular idea but its deeply flawed and should be avoided at all costs.

Separate these things in your pipelines:

  1. building your code
  2. creating a container image with the compiled code in
  3. publishing that image to a container repository
  4. deploying containers using the image from the repository

This gives you the flexibility you are looking for as you now have multiple points where you can test your compiled code and stop the pipelines while avoiding unwanted artefacts.

1.5 test the code before creating the image.

since you have the compiled code before you have even started making an image, you can spin it up, or dynamically link to it and run any tests you want.

Failing here means an image is never created

2.5 test after you have created an image but before its published.

You have an image with the compiled code, but you are still in a build pipeline. spin up a container using the local image and run tests against it

Failing here means the image is only ever local to your build pipeline and will presumably be destroyed when the pipeline completes.

3.5 You create the image and publish it to your repository, but its still not deployed.

You can deploy the image to a test environment and run tests against it

If it fails here you have published image in your repository, but you have not deployed it to live. Perhaps you could have two repositories and only promote to the live one if the tests pass

2
  • Hi can you elaborate further "your problem stems from the docker philosophy of compiling your code in a docker file" ? Why should it be avoided at all costs? I was confused about that, e.g. I use typescript, so in my dockerfile I have a "compile" step to transpile the ts code into js. Also my question is about what is the problem of adding test step in dockerfile? Mar 25, 2022 at 1:33
  • you go into some of the problems in your question. essentially you end up programming in dockerfile commands, which is extremely limiting.
    – Ewan
    Mar 25, 2022 at 17:26
1

As Hans-Martin Mosner said in his comments that the problem with my proposed process is dirty.

First, because I wanted to make sure my node app can start correctly, I didn't put npm start in background, I then needed to add some check in my startup logic that if this is a docker test run, exit the process after some specific times. Second, because npm start didn't exit I then need to add codes to run test after npm start. So my source codes and package.json both add ugly codes for the docker test run.

Because my goal is to reduce failed docker images, the better way to do it as he pointed out is to only push the docker image to my registry when it passes the test and delete failed image on local build server.

And I find GitHub action is a perfect place run my test. I am not sure if GitHub action using docker or not, but what I need is a clean environment to run/test my app and GitHub action fits the bill. So my CI process changes to only when I pass GitHub action I will build docker image and push it to my registry. Actually if GitHub action can push docker images directly to my registry I will only it then.

3
  • That looks like a sound solution (I'm not familiar with GitHub actions but that apparently is a good use for them). If I understand correctly, you're doing a bit of what @Ewan proposed: Compile and test your code (using GitHub actions) before starting to actually build a Docker image. There is still a possibility that the final Docker image isn't ok, so integration testing against it still makes sense, but you'll get much fewer failed images in your registry indeed. Mar 30, 2022 at 7:54
  • Your understanding is correct, compile & test my code using GitHub action. And yes it is still possible the final docker image is not ok. Mar 30, 2022 at 7:59
  • Let GitHub action run my docker image and test against it is bit tricky. I am still learning and testing it. Mar 30, 2022 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.