Storing both a compiled service for a Docker container as well as the container in a repository. Good practice, bad practice or no precedent?

Question

This has erupted from quite a turbulent meeting between two senior developers, a lead developer and an engineering lead, and after 90mins reached no resolution.

We create Spring Boot Java services which are then dockerised before being deployed to an environment.

As part of our automated build our services are compiled via Maven, resulting in a jar, which then gets pushed to the package registry within gitlab. We then containerise this jar (we don't obtain via the package registry, it just receive it via previous stages in the pipeline) using Docker, which we then push to the container repository.

This means for our service we have the Docker container available, as well as the jar. Note, we don't dockerise anything else, and configuration is held within these jars.

Points raised were following.

• Storing both the jar and the container takes up space. This was the point everyone agreed on. It does. However we don't have limits on space we can take up within our organisation and can easily set up expiries only keeping certain jars, e.g. latest within the package registry.
• We have access to the Docker container. Why do we need the jar as well. This was a VERY contentious point. With arguments being made if an issue arised and you needed access to the jar, maybe to inspect it's contents such as configuration, it's much easier to download the jar, rather than obtain it via the Docker container. Counter point to that was you should be able to rebuild the application at a certain point, this is why tagging exists. Counter counter point. Builds take time. More importantly what you build locally you can't guarantee will be the same as what has been built by CI. Different Java versions and implementations to begin with (Azul vs Coretto, 17.0.1 vs 17.0.2) not to mention architecture (we're using a mix of Intel and M1 Macs which alone has caused issues) doesn't guarantee it will be built the same. There was further disagreement in that Java doesn't work this way, ending in 'yes it does' vs 'no it doesn't'. Locally it's worth noting we don't rebuild the application against a version number, we just use 'local' (mvn version is used prior to building on master to properly set the version within our apps).
• It's not a package. We deploy Java libraries and DTOs to our package registry mainly to make them available to other projects via the package registry. This is the way we use it for the main. Chances are extremely low (unless we make something that requires the jar, such as a new testing thing) that we will need to obtain the service like we typically do with DTOs and other libraries.
• Outside of our team. We can't see anyone else is doing this. This sets some organisation precendent. However engineering practices tend to get set on a team by team basis, such as tools/languages/coding/dev standards. And no other teams have a deployment pipeline quite like ours. But there doesn't seem to be a defined rule on this.
• Doing this is 'Bad Practice'. There appears to be nothing online either way about this being good or bad practice. Hence the question is being asked on here.

Ultimately in theory you might want to have the jar available, in practice, you may never need to. But:

It's better to have it and not need it. Then need it and not have it.

So ultimately the question is this. Is storing both our Jar and container good, bad, or actually there's no engineering precedent for this. It's down to your organisation. Are there big arguments either way which have been missed. I'm however scared of this being a discussion, which I presume are off topic here, hence why I think this is an option.

Full disclaimer. I am in the camp of why are we spending so long arguing about this. Does it really matter. There are reasons we may want to access the jar rather than recompiling it (which may not be the same as that produced on CI), and storage is cheap. I don't think it's practice either way. I have in fact had to download a jar to check a version number before. This version number can be obtained via the running container, however to run the container it required a bunch of other containers, and was quicker and easier to just check the file within the jar.

Answer 1

Some random thoughts:

1. Has a saved JAR ever been used in your organization for any useful purpose, even once? Did the cost of storing the JAR exceed the benefit of having it available?

2. Is there an identified need of any kind, potential or actual, for a binary snapshot? Given the environmental complexities, I don't see how you guarantee an identical binary without storing a JAR. There are just too many variables to consider, and it's almost certainly cheaper just to store it than it is to attempt to recreate the exact environmental conditions under which it was originally created.

3. Is there really such a thing as a "best practice," especially one that wins a popularity contest? If it provides benefits to your organization at very low cost, does it really matter what anyone else thinks?

Answer 2

Though I am not a Docker expert, I have been in a very similar situation in the past, just with different packaging technologies. The question was "Once we packaged our software and deployed it, and something goes wrong afterwards, how easy or hard is it to retrieve the original package content again" (either for debugging, inspection or repackaging purposes).

Hence, I think the crucial point is what you wrote here:

maybe to inspect it's contents such as configuration, it's much easier to download the jar, rather than obtain it via the Docker container.

If this is definitely the case and cannot be changed, you better keep both, the Docker file as well as the jar. However, you should ask yourself why obtaining the jar from the Docker container is so much harder. If you can add some scripts or tools (or maybe a simple "readme" file) to make this step easier, then there will probably no need to store both. Storing as few data redundantly as possible is often beneficial, since it eliminates a potential source of errors. I found this highly upvoted Stackoverflow Q&A from 2014 on how to extract the build artifacts out of a Docker container again, maybe you find something useful among the answers.

Of course, this may be just a performance issue: if running a script to reach out to the content of the Docker container runs 15 minutes, and just downloading the jar takes 15 seconds, you may be inclined to keep the jar file (though 15 minutes may be fast enough if you expect this to happen only once a year).

And of course, as others have already written, I would not rely on being able to reproduce the very same jar file from source by a rebuild. The build process itself could contain some error, or dependend on some unforeseen contextual influences. Sooner or later you need to have a way to get exactly the jars which were deployed into production. So either keep them or create a simple way to extract them from what you deployed.

Answer 3

JARs and Docker images have different purpose. That's I would keep both.

If you have a JAR, it has following advantages:

1. You can run it on any machine, even if Docker is not installed.
2. It is much easier to inspect its contents, configuration files inside, particular libraries used.
3. It is easier to compare different versions by their JARs, not by Docker images.
4. Rebuilding from source produces the same JAR only in ideal case. In reality, the can be differences. Some reasons can be following. A) You have mistakenly released some module JAR with different contents, but with the same version number, and the earlier version was packaged to your application JAR. When you build it later on, it will take the newer module JAR and the application JAR will differ. B) Or there was some infrastructure problem and the build machine used wrong branch or wrong tag or checkout was not successful. If you rebuild this tag, you will not reproduce the application JAR. Having a previously built JAR will help you to understand what exactly was built.
5. It allows you to build Docker image using a newer version of base image and an older version of the JAR.

TLDR: I would keep both, JAR and Docker image.

Answer 4

"good practice/bad practice" is basically delegating your thought process about what is best for you to others. (or it is a "it is more than just me" argument).

If what you have works for you, then why change?

If the jar file in itself is just a temporary thing and the docker image is the real product of your workflow, and your IDE can produce the exact same sources (this helps a lot if you save the git commit information in the image) there is no reason to save it as such.

Record the information you need at debug time in the docker image. Learn to use a debugger that can reach inside the docker image. Do this ahead of when you need it.

And have a CTO that can make these decisions for your team, or the arguing will never stop.