1

I'm interested in modernizing an existing codebase for a commercial software package written in Java, in part by moving it to Maven. The primary motivator here is to shrink our git repo, which is very large due to the build system being ANT with its dependencies checked into git. I've done a few small side projects in Maven to learn the ropes, but there's a big piece of knowledge that I'm missing before I can convert the big project, and can't seem to search up any answers.

Our release build process generates an installer, and we have some large non-Java binaries that need to be available at build time so that they can be bundled into our installer (they are installers for various frameworks our software interacts with at run time). Presently, those installers are just part of our git repo, which is part of what is making it so large. What is the proper way to manage such files with Maven so that they can be fetched at build time if they're not present?

Should we host the files on some kind of internal webserver and have a maven plugin fetch them by URL? Can this sort of thing be handled by private Maven repos, even though the files in questions aren't Java code, and aren't proper Java dependencies in the sense that they won't end up inside the compiled JARs, but just need to be present at build time for installer-bundling? I'd love to know what the standard way to solve this issue is, but the difficulty of finding an answer makes me wonder if I'm asking the wrong question.

2
  • The answer of what to do with them really depends on the exact nature of those files and what the requirement is for them (are they only needed in the installer and not by developers for local dev/testing? What kinds of dependencies are they specifically who owns or controls their content? Do they originate from a third-party? Would they ever be modified as part of the build process? Would they ever be modified during installation? would the application ever modify them at runtime? etc.) Jul 15, 2022 at 8:07
  • @BenCottrell As a practical example, let's say that we wanted to bundle the installer for Amazon Corretto, so that our installer could install a JVM if one wasn't present. In this case, it is an .msi file, originating from a third party, not needed for dev/testing, not modified at build/install/run times (merely executed by our installer at install time). I'm looking for the right way to automate the download/inclusion of this .msi file without bloating our git repo with it. Thank you, and I would love to hear your thoughts. Jul 15, 2022 at 16:46

2 Answers 2

0

You are quite correct that these files clog up a source control system. I would recommend you set up a server (preferably running a Linux distro) accessible to all your developers, and install a Maven mirroring application like Nexus. This will give you:

  1. A place to store large binaries that seldom change.
  2. A place to store the artifacts produced by your build processes
  3. A Maven repository mirror containing only the artifacts that you use.

Obviously you still need lots of disk space, but the git repository can be shrunk considerably (not my area of expertise).

I also suggest you look at running your build scripts from a tool like Jenkins. This provides:

  1. A nice UI, that will prompt for the required parameters the scripts need.
  2. Version management. The master POM file will have its version automatically incremented after a build, ready for the next build.
  3. A history of what was built and when.
  4. Automatic tagging of the git repository files that were used by the build.
  5. A central place for all your build scripts. For instance it is quite possible to have jobs that consist of just OS commands - e.g. FTP a file to test server and restart Tomcat.

Most of our Jenkins jobs are quite simple - they do a little configuration then run a maven command to do the actual build.

Getting all this running can be a bit of a learning process, so if you can find a contractor with the right skills, a day or so of their time will soon pay for itself.

We have found that you don't need Nexus or Jenkins on development machines. The modern IDEs run Maven scripts without external support.

Best of luck - having a build system that is predictable and easy to use is a great productivity booster.

1
  • Thank you very much for all of this information! I did not realize that Maven repos could contain arbitrary artifacts, and I was stuck on the idea that Maven was for providing me JAR-based Java dependencies for inclusion in my generated artifacts. It sounds like my answer is to set up a private Maven repo for my chunky support binaries, have my Maven-aware project ask for Maven to fetch them, and then use them as I will during the build process. Thanks also for the well wishes. I know this will be a big task, but I'm glad that I now know the correct place to start. Jul 18, 2022 at 16:39
0

Note: I won't go into any Maven tool specifics, firstly on the basis that I'm not all that familiar with Maven itself, and secondly that questions about how to use particular tools are off-topic and belong on StackOverflow.

External Dependencies

In general (and the case for Maven), most package/artefact services and tools are nothing more than generic repositories which are agnostic to the content of the files they serve, but whose tools also happen to have additional support which lend themselves to a particular programming language.

With that in mind, Maven along with most other similar tools are well-suited for serving generic, packed archive files such as .zip or .tar, making them convenient 'mirrors' for 3rd party dependencies.

Related question on StackOverflow: https://stackoverflow.com/questions/25441633/how-to-deploy-only-zip-artifacts-in-maven

Using a package repository as a mirror provides a safety net against problems with external sources, including availability, the chance that any 3rd party might make a change which breaks your build process, and helps development teams unify all of their devops processes around a single tool, so I would consider the use of a private package repository for this purpose as a fairly standard approach.

I'd suggest unifying around a single portable archive format such as .zip for all 3rd-party dependencies. Even if the particular file you want to host uses some other archive format such as MSI, it should be trivial to put an MSI inside a zip when publishing it, and then extracting it as part of the build process. All consumers of all 3rd-party artefacts/dependencies can then rely on a single consistent method for retrieving and unpacking those dependencies.

Settling on an archive format such as .zip also solves any problems if other files need to be included - for example, you might want to ensure all 3rd-party packages contain some setup/deploy script and a readme.

Git Cleanup

Given that these large files are currently stored in git, and that deleting a file from git doesn't erase git history, the git repositories won't shrink just by moving those files out, some invasive surgery will be needed: https://stackoverflow.com/questions/2100907/how-to-remove-delete-a-large-file-from-commit-history-in-the-git-repository

I would recommend a great deal of caution and coordination with everyone who uses these git projects. Consider forking the git repository, making the original repository read-only, asking developers to move to the new fork, so that any erased history isn't lost forever in case anything goes wrong. You may wish to keep the original read-only repository hanging around for as long as it contains any historic tags or release branches which still need to be supported.

To verify that nothing has broken, clone both the original and changed repositories side-by-side and use HEAD commits in each of your main/master branches to ensure there are no differences; remember to also check other long-lived branches (if applicable) - for example, any current/recent release branches, and any active development branches. (Ideally arrange to do the work when there are as few active branches as possible). This may seem over-cautious, but it should be a one-time deal for each repository.

1
  • Thank for very much for all this! I was fixated on the apparently mistaken idea that Maven was for storing JAR-based Java dependencies. Knowing that I can use it for arbitrary binaries for my build process is a huge help. Thanks for the warning about git repo size, as well. I knew to expect this, and plan to either strip historical large files, or just restart the history when we migrate to cloud-hosted git repos from our self-hosted repos, and archive off the deep history in cold storage. There's a long road ahead, but I now know where to start. Jul 18, 2022 at 16:43

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .