How do you approach transitive dependency conflicts that are only known at run-time? [closed]

Question

How do you normally approach transitive dependency issues that occur at run-time in large software projects?

For the past three weeks, I have been trying to start a component of a large piece of software within another component of the software, but it intermittently dies due to transitive dependency issues that are only known at run-time.

By transitive dependency issues, I mean that certain dependencies of a given project's dependencies collide with other dependencies at run-time, causing instability, or instant failure.

There are hundreds, upon hundreds of dependencies in use, and there are approximately 50 sub-projects associated with the tool that are worked on in isolation by other teams, where all modules have deeply nested dependencies between each other. Nobody knows what all of the sub-projects are used for, given the scale, and complexity of the project.

In this situation, would you try to generate a visual representation of the DAG for each of the dependencies of the affected component, and attempt to determine where collisions may occur at run-time? I have no control over how dependencies are managed in other sub-projects, and cannot change any Java code that has been written by other developers

The solutions I've come up with only work for an hour, or two, and then they stop working due to changes in upstream components. An example of an upstream component is an artifact on which the project that I am working on is dependent on which is built at an earlier stage in the CI pipeline.

---

At other's requests, I am going to include information on what technology is being used, at the risk of having the question closed for providing too much information, or the body getting too long:

• Maven is used for dependency management; and
• Spring is used as a DI container;
• Most of the dependency issues involve overlapping bean contexts as a result of the contexts of other modules being loaded at run-time
• The product works properly, and there are smorgasbords of unit tests, and integration tests to elude to the functional correctness of the program

In general, I am looking for a language-agnostic approach to identifying ways of resolving dependency conflicts without enumerating through all possible combinations of a given project's dependencies.

I cannot re-architect the project, add additional quality gates, push for paradigm shifts across the company, or switch languages as a resolution.

Answer 1

It may be possible to solve your problem using custom classloaders to give each module in the application its own execution environment.

Basically, you would define a core environment that consists of a small kernel of the application, along with whatever dependencies are universally agreed on and all the interfaces that are necessary for modules to communicate with each other. Then, each module you load gets its own classloader that can access (1) classes from the runtime environment, (2) classes from the application core, and (3) classes from that module and its direct dependencies. All inter-module communication goes through interfaces defined in the core so that no module directly depends on another.

This technique is used in application servers to allow applications to have dependencies that would otherwise clash, so you can see a working implementation of the technique, for example, in Tomcat (and, if you're lucky, you may be able to use Tomcat's implementation with little change).

Answer 2

I have never worked with Maven or Spring, but to give you a generic answer to a generic question: when it comes to dependency issues, your system should be designed to detect them at the earliest possible point in time, and when those issues are detected, they should be signaled immediately, including their possible cause.

Ideally, this point in time is not at "production run time". The best point in time is "build time", but that seems to be impossible in your case. So the second best option are run time tests. You told us there are "integration tests", but as they do not detect the dependency issues, they seem to be incomplete, or they do not test dependency collisions in general. That's where you should start trying to solve the problem.

Moreover, to make the tests more effective, your components need to "fail fast" when a dependency issue arises. That means, as soon as a dependency is resolved and a component is loaded, potential dependency collisions should be checked and signaled immediately. If the system first runs for an hour before a collision will be detected, it becomes very hard to spot the cause for the problem. I do not know if Maven/Spring already provides you with an in-built "fail fast" strategy, but if this is not the case, try to think in that direction.

To mitigate problems in production, your system should be designed not to die completely when a dependency collision occurs with a less important sub-component involved. The failure should not be masked, of course, but the system should ideally reject the loading of that component, log and/or signal the problem and stay in a stable state. When the component is loaded first, the system becomes unstable and you detect the issue only afterwards, then you will probably have to shutdown and restart the system completely, which I guess is something you better want to avoid.

For example, if your system is a Web shop, and your submodule for "newsletter subscription" cannot be loaded for some hours, that's something which might be easier tolerated as if the whole shop does not work any more.

Finally, this might not be an option in your case, but if components can be made run in better isolation against each other, and if that can avoid dependency collisions, that may be an approach worth to think about.

Answer 3

Consider this me just thinking out loud here. Depending on requirements/time constraints I might consider this:

1) create a list of interfaces and their implementations (using reflection, if available)

2) create a sort of wrapper around your DI. When a DI is asked to provide concrete implementation, see if there's a DI rule already in place; if rule exists - just return what your DI provides; otherwise if there's a single implementation of an interface and you're able to create a instance - log, and return an instance; otherwise log and fail.

Depends on how involved you want to get, I guess, but eventually you just want to have a complete list of what requires what - this solution will eventually generate that list.

However, arguably this is a lot of work and is not reliable - what if you have an odd condition requiring certain dependency once in a blue moon?

What do you mean when you say "changes in upstream components" ?

Answer 4

If I follow you correctly, the issue is that the code you need to use has run-time dependencies on modules expressed in Spring XML, but maven is not told about them. So when you use them the things they need (the transitive dependencies) are not on your classpath, and they don't do what they should.

I'm guessing when you ask your colleagues 'how do i get this to work?', they say 'obviously you need this list of 35 modules and versions, how could you not know that?'

Presumably when integration testing is done, the required modules are declared as test dependencies. So the systematic solution is to set up inspections of integration test pom's to ensure they have nothing but 3rd-party test tools declared as test dependencies. This could even be automated by the maven enforcer plugin.

Apparently you can't do that, so your best option is likely found here.