Assume, I have a relatively large and complex application (100 MB .war file) with multiple dependencies. Some part of the code is defining an object factory, where similar objects (all inheriting from the same base class) are instantiated.

My goal is to convert these hard coded objects to plugins that can be loaded dynamically at run-time.

So, I wrote a plugin loader:

String localPath = "...";
String pluginName = "...";

File jarFile = new File(localPath);
ClassLoader pluginLoader = URLClassLoader.newInstance(new URL[]{jarFile.toURL()});

pluginLoader.loadClass(pluginName).newInstance();

I then went on to actually write the plugins, which is when I realized that each potential plugin has a large number of dependencies and creating a jar file for each of them would result in plugins of roughly 50MB in size.

I'm worried about the memory requirements this would entail, when loading 100 of these plugins.

Is there a way I can load plugins that depend on classes inside the main app without bundling them into the plugin itself?

Or how would I best share such common resources?

[UPDATE] For now I figured I could expose the already loaded classes to the ClassLoader, which I was told works like this:

ClassLoader mainLoader = PluginManager.class.getClassLoader();`
ClassLoader pluginLoader = URLClassLoader.newInstance(
                                 new URL[]{jarFile.toURL()},
                                 mainLoader);

The downside to this is obviously that any change in the main application can make the plugins fail. This will, however, help with my memory concerns and with proper error handling it will be easy to find no longer working plugins.

  • 1
    It sounds like you want OSGi. Or Some design pattern involving singletons. – Richard May 8 '17 at 12:28
  • I don't get it. Why do you want to load them at runtime? Why not develop them in a separate project/artifact and declare it as a dependency? That would be trivial and efficient since the build tool will take care of dependencies checking and deduplication. – dagnelies May 11 '17 at 14:54
  • Because it's my requirement - that's the whole point of using plugins in the first place. – Chris May 11 '17 at 22:24
  • @Chris: "My goal is to convert these hard coded objects to plugins that can be loaded dynamically at run-time." That's not a requirement. That's how you want to achieve something, and that's perhaps less than ideal. See XY problems. The question is still: why in a second step at runtime? Why not directly as dependencies? – dagnelies May 12 '17 at 14:55

First, I should point out that external dependencies (not from source code of your project or it's subprojects, in Gradle terms) are not included into jars, unless you specifically create a "fat" jar (or "uberjar").


Probably you're looking for compileOnly dependencies (in Gradle terms) or provided dependencies (in Maven terms). Such dependencies are only used to compile your project, but are not included into resulting distribution.

The idea is that, in a plugin code, you specify your main project as a compileOnly dependency, so that you can implement interfaces from the inside of it.

Compare compileOnly with runtime:

  • runtime: "these need to be there when this program executes, so that my code can use instances of them"
  • compileOnly: "those are just needed for type-checking to compile this code, but it doesn't matter if they exist on runtime"

Since you only need some API interfaces to implement, the latter is your case.

  • This doesn't seem to answer the question, which is how to arrange for compile-time dependencies to be available at run time when the jar is loaded. – Jules May 7 '17 at 21:05
  • No, I think you misunderstood the question. It is "how do I depend on app interfaces inside a plugin without bundling them into it," which I answered clearly. As for compile-time deps, they will be available (and provided to a plugin) in the running app by it's definition, there is nothing to arrange. – scriptin May 7 '17 at 21:41
  • quoting from the question with emphasis: "Is there a way I can load plugins that depend on classes inside the main app without bundling them into the plugin itself". It doesn't mention anything at all about how to build such a plugin, which I presume the poster already knew how to do, just how to load them. – Jules May 8 '17 at 7:34
  • Well, your presumption seems to be wrong as well, because clearly poster is concerned about the size of plugins' jars which he's going to build. It suggests that he haven't already, and looking for a way of keeping them small by not including dependencies from the main project. – scriptin May 8 '17 at 8:33
  • @Jules I think this is ultimately does address the OPs issue but I think in order to help the OP, there needs to be more detail. – JimmyJames May 9 '17 at 15:20

Delaying of loading of the dependencies will not solve the memory problem; it will just delay it.

100MB war file is not really that big (depending on your server) and not everything is loaded into memory on startup. Classes are loaded into memory only if the instances/statics are created. If there is too much memory required, there are too many instances, not too many classes. I would suggest to look for a pooling solution for those object factories.

  • The problem is that I would have 100 such plugins - each claiming 100MB will add up – Chris May 11 '17 at 22:23
  1. Identify the internal/external dependencies of the plugin, review them and create a jar out of the dependencies, Hope it gives an opportunity to refactor as well. There has to be something which is cohesive about it. This needs to be managed a separate project with its version. (Needed to track them with separate versions.)
  2. Include this a compileOnly in case of plugin and include the jar in the main application.
  3. Dynamic loading of the plugin in the main application as done in the application.

What you are describing is already handled by Java and web containers will often add more complex class loading (you are using a war file.)

Let's start with the basic class loading process (oversimplified.) In Java if you try to load a class or use a dependency, it will search the classpath for that dependency. Whatever instance of that dependency is found first, get's pulled in. So if I have two classes in separate jars that both depend on the same Logger class from Log4J, I don't need two copies of Log4J in my classpath. Only one will ever get used. But there's a problem here. What if one of your jars needs version 1 of com.foo.bar.Snafu and another one needs version 2 of com.foo.bar.Snafu? This simple model won't work for you.

Because the idea behind JEE is that you have a lot of different things running all together with an bazillion dependencies, this comes up a lot. So in my experience (which is not recent - more on that later) most containers offer a more complex classloading model. The first step is to search (recursively) through all parent class loaders for a class. If it is not found, it will be loaded from the web apps local class path. So if you want to have one copy sof Log4J for everyone, you put it on the containers main classpath. If everyone can use that version, you are done. Your wars or ears need not provide binaries for that dependency.

However, if you need different versions for different web apps under this model, that means you need to get it out of the main classpath and add the different versions at different levels of the classpath hierarchy. This is complex and fraught so containers offer the ability to flip that model on it's head in your web app. That is, it will look first in it's local classpath before looking in parent classpaths.

So regardless of which way you go, if you want to share these dependencies, just put them on the classpath at the appropriate level. If they are used by multiple apps, consider putting them at the root.

Another thing to consider is the fact that all of this nonsense kind of sucks to work with. I highly encourage anyone doing web development in Java to consider embedded approaches such as embedding Jetty or embedding Tomcat. It will make your life so much easier and it is much more compatible with containerization.

P.S. One of the comments mentions OSGi. This does, in theory, address a lot of these issues. However, I would not recommend this unless you are a masochist. It makes things far more complicated and won't necessarily work for you.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.