How do you normally approach transitive dependency issues that occur at run-time in large software projects?

For the past three weeks, I have been trying to start a component of a large piece of software within another component of the software, but it intermittently dies due to transitive dependency issues that are only known at run-time.

By transitive dependency issues, I mean that certain dependencies of a given project's dependencies collide with other dependencies at run-time, causing instability, or instant failure.

There are hundreds, upon hundreds of dependencies in use, and there are approximately 50 sub-projects associated with the tool that are worked on in isolation by other teams, where all modules have deeply nested dependencies between each other. Nobody knows what all of the sub-projects are used for, given the scale, and complexity of the project.

In this situation, would you try to generate a visual representation of the DAG for each of the dependencies of the affected component, and attempt to determine where collisions may occur at run-time? I have no control over how dependencies are managed in other sub-projects, and cannot change any Java code that has been written by other developers

The solutions I've come up with only work for an hour, or two, and then they stop working due to changes in upstream components. An example of an upstream component is an artifact on which the project that I am working on is dependent on which is built at an earlier stage in the CI pipeline.


At other's requests, I am going to include information on what technology is being used, at the risk of having the question closed for providing too much information, or the body getting too long:

  • Maven is used for dependency management; and
  • Spring is used as a DI container;
  • Most of the dependency issues involve overlapping bean contexts as a result of the contexts of other modules being loaded at run-time
  • The product works properly, and there are smorgasbords of unit tests, and integration tests to elude to the functional correctness of the program

In general, I am looking for a language-agnostic approach to identifying ways of resolving dependency conflicts without enumerating through all possible combinations of a given project's dependencies.

I cannot re-architect the project, add additional quality gates, push for paradigm shifts across the company, or switch languages as a resolution.

closed as too broad by BЈовић, durron597, GlenH7, Ixrec, Eric King Aug 31 '15 at 23:36

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Are you talking about something like the DI container isn't properly setup to know that for IFoo use IFooImpl? – Andy Aug 24 '15 at 20:20
  • No, it's more like package (b) in project (b) is a transitive dependency of package (a) in project (A), but (B) is only introduced at run-time. The software all works from what I can tell, but I just need to appropriate set of dependencies to allow for all of the transitive dependencies to be resolved properly. There are sets of dependencies that will allow the product to start-up properly, but, those configurations are very brittle. As a junior dev. I have no control over this. – Tyler Aug 24 '15 at 20:29
  • 2
    Come on, "dependency issue" is a very generic term, and everybody who reads this probably envisions something different. For real-world softare this is technology dependend, so please give a real example for a specific technology you have in mind. Are the issues are arising because they needed components are missing? Or because a wrong version is provided? What kind of dependency resolution mechanism is already in place? Please clarify. – Doc Brown Aug 24 '15 at 20:56
  • 1
    In the .Net world I solve this problem just be installing the packages into my main application project. While no application code actually uses the plugin libraries directly the .Net compiler ensures the DLL is in the right spot during build and thus deployment. – Andy Aug 25 '15 at 12:50
  • 2
    " I have no control over how dependencies are managed, and cannot change any code" - what actually can you change? If you cannot change anything, the question seems pointless. – Doc Brown Aug 25 '15 at 14:02

It may be possible to solve your problem using custom classloaders to give each module in the application its own execution environment.

Basically, you would define a core environment that consists of a small kernel of the application, along with whatever dependencies are universally agreed on and all the interfaces that are necessary for modules to communicate with each other. Then, each module you load gets its own classloader that can access (1) classes from the runtime environment, (2) classes from the application core, and (3) classes from that module and its direct dependencies. All inter-module communication goes through interfaces defined in the core so that no module directly depends on another.

This technique is used in application servers to allow applications to have dependencies that would otherwise clash, so you can see a working implementation of the technique, for example, in Tomcat (and, if you're lucky, you may be able to use Tomcat's implementation with little change).

  • 1
    I'm not permitted to re-architect the project to solve the dependency issue. Switching to a microkernel-like architecture would be kind of cool, though. – Tyler Aug 25 '15 at 14:44
  • 2
    @Tyler: nevertheless I think this is a good, generic answer to the generic part of the question. – Doc Brown Aug 25 '15 at 14:46
  • 1
    That sounds a bit like osgi. – SpaceTrucker Aug 26 '15 at 7:11

I have never worked with Maven or Spring, but to give you a generic answer to a generic question: when it comes to dependency issues, your system should be designed to detect them at the earliest possible point in time, and when those issues are detected, they should be signaled immediately, including their possible cause.

Ideally, this point in time is not at "production run time". The best point in time is "build time", but that seems to be impossible in your case. So the second best option are run time tests. You told us there are "integration tests", but as they do not detect the dependency issues, they seem to be incomplete, or they do not test dependency collisions in general. That's where you should start trying to solve the problem.

Moreover, to make the tests more effective, your components need to "fail fast" when a dependency issue arises. That means, as soon as a dependency is resolved and a component is loaded, potential dependency collisions should be checked and signaled immediately. If the system first runs for an hour before a collision will be detected, it becomes very hard to spot the cause for the problem. I do not know if Maven/Spring already provides you with an in-built "fail fast" strategy, but if this is not the case, try to think in that direction.

To mitigate problems in production, your system should be designed not to die completely when a dependency collision occurs with a less important sub-component involved. The failure should not be masked, of course, but the system should ideally reject the loading of that component, log and/or signal the problem and stay in a stable state. When the component is loaded first, the system becomes unstable and you detect the issue only afterwards, then you will probably have to shutdown and restart the system completely, which I guess is something you better want to avoid.

For example, if your system is a Web shop, and your submodule for "newsletter subscription" cannot be loaded for some hours, that's something which might be easier tolerated as if the whole shop does not work any more.

Finally, this might not be an option in your case, but if components can be made run in better isolation against each other, and if that can avoid dependency collisions, that may be an approach worth to think about.

Consider this me just thinking out loud here. Depending on requirements/time constraints I might consider this:

1) create a list of interfaces and their implementations (using reflection, if available)

2) create a sort of wrapper around your DI. When a DI is asked to provide concrete implementation, see if there's a DI rule already in place; if rule exists - just return what your DI provides; otherwise if there's a single implementation of an interface and you're able to create a instance - log, and return an instance; otherwise log and fail.

Depends on how involved you want to get, I guess, but eventually you just want to have a complete list of what requires what - this solution will eventually generate that list.

However, arguably this is a lot of work and is not reliable - what if you have an odd condition requiring certain dependency once in a blue moon?

What do you mean when you say "changes in upstream components" ?

  • Sorry, I didn't want to provide too much background information - when I describe the operating environment, questions are usually downvoted, and flamed to death. I'll update the original question with a definitions section. – Tyler Aug 25 '15 at 14:14

If I follow you correctly, the issue is that the code you need to use has run-time dependencies on modules expressed in Spring XML, but maven is not told about them. So when you use them the things they need (the transitive dependencies) are not on your classpath, and they don't do what they should.

I'm guessing when you ask your colleagues 'how do i get this to work?', they say 'obviously you need this list of 35 modules and versions, how could you not know that?'

Presumably when integration testing is done, the required modules are declared as test dependencies. So the systematic solution is to set up inspections of integration test pom's to ensure they have nothing but 3rd-party test tools declared as test dependencies. This could even be automated by the maven enforcer plugin.

Apparently you can't do that, so your best option is likely found here.

  • That's exactly what's happening, from what I can tell. So far I'm juggling 30 dependencies, and twiddling the scopes to eliminate collisions, but, peer review is what is needed here. – Tyler Aug 26 '15 at 14:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.