JVM developer here. Lately I've seen banter on IRC chat rooms and even in my own office about so-called "shaded" Java libraries. The context of the use will be something like:
"Such and so provides a "shaded" client for XYZ."
Perfect example is this Jira issue for HBase: "Publish a client artifact with shaded dependencies"
So I ask: What is a shaded JAR, what does it mean to be "shaded"?
Shading dependencies is the process of including and renaming dependencies (thus relocating the classes & rewriting affected bytecode & resources) to create a private copy that you bundle alongside your own code.
The concept is usually associated with uber-jars (aka fat jars).
There is some confusion about the term, because of maven shade plugin, which under that single name does 2 things (quoting their own page):
This plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade - i.e. rename - the packages of some of the dependencies.
So the shading part is actually optional: the plugin allows to include dependencies in your jar (fat jar), and optionally rename (shade) dependencies.
Adding another source:
To Shade a library is to take the contents files of said library, put them in your own jar, and change their package. This is different from packaging which is simply shipping the libraries files in side your own jar without relocating them to a different package.
Technically speaking, dependencies are shaded. But it's common to refer to a fat-jar-with-shaded-dependencies as "shaded jar", and if that jar is a client for another system, it can be referred to as "shaded client".
Here is the title of the Jira issue for HBase that you linked in your question:
Publish a client artifact with shaded dependencies
So in this post I'm trying to present the 2 concepts without conflating them.
Uber-jars are often used to ship an application as a single file (makes it easy to deploy and run). They can also be used to ship libraries along with some (or all) of their dependencies shaded, in order to avoid conflicts when used by other applications (which might use different versions of those libraries).
There are several ways to build uber-jars, but maven-shade-plugin goes one step further with its class relocation feature:
If the uber JAR is reused as a dependency of some other project, directly including classes from the artifact's dependencies in the uber JAR can cause class loading conflicts due to duplicate classes on the class path. To address this issue, one can relocate the classes which get included in the shaded artifact in order to create a private copy of their bytecode.
(Historical note: Jar Jar Links offered that relocation feature before)
So with this you can make your library dependencies an implementation detail, unless you expose classes from those libraries in your API.
Let's say I have a project, ACME Quantanizer™, which provides DecayingSyncQuantanizer class, and depends on Apache commons-rng (because of course to properly quantanize you need a XorShift1024Star, duh).
If I use the shade maven plugin to produce a uber-jar, and I look inside, I see these class files:
com/acme/DecayingSyncQuantanizer.class
org/apache/commons/rng/RandomProviderState.class
org/apache/commons/rng/RestorableUniformRandomProvider.class
...
org/apache/commons/rng/core/source64/XorShift1024Star.class
org/apache/commons/rng/core/util/NumberFactory.class
Now if I use the class-relocating feature:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>com.acme.shaded.apachecommons</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
The content of uber-jar looks like this:
com/acme/DecayingSyncQuantanizer.class
com/acme/shaded/apachecommons/rng/RandomProviderState.class
com/acme/shaded/apachecommons/rng/RestorableUniformRandomProvider.class
...
com/acme/shaded/apachecommons/rng/core/source64/XorShift1024Star.class
com/acme/shaded/apachecommons/rng/core/util/NumberFactory.class
It's not just renaming files, it rewrites bytecode that references relocated classes (so, my own classes & commons-rng classes are all transformed).
In addition, Shade plugin will also generate a new POM (dependency-reduced-pom.xml) wherein shaded dependencies are removed from the <dependencies> section. This helps use the shaded jar as a dependency for another project. So you can publish that jar instead of the base one, or both (using a qualifier for the shaded jar).
So that can be very useful...
...but it also poses a number of issues. Aggregating all dependencies into a single "namespace" within the jar can get messy, and require shading & messing with resources.
For example: how to deal with resource files that include class or package names? Resource files such as service provider descriptors which all live under META-INF/services?
The shade plugin offers resource transformers that can help with that:
Aggregating classes/resources from several artifacts into one uber JAR is straight forward as long as there is no overlap. Otherwise, some kind of logic to merge resources from several JARs is required. This is where resource transformers kick in.
But it's still messy, and the problems are almost impossible to anticipate (quite often you discover the issues the hard way in production). See why-we-stopped-building-fat-jars.
All in all, deploying a fat jar as a standalone app/service is still very common, you just need to be aware of the gotchas, and for some of those you might need shading or other tricks.
There are many more difficult issues (debugging, testability, compatibility with OSGi & exotic classloaders...).
But more importantly, when you produce a library, the various issues that you thought you could control are now getting infinitely more complicated, because your jar will be used in many different contexts (unlike a fat jar that you deploy as a standalone app/service in a controlled environment).
For example, ElasticSearch used to shade some dependencies in the jars they shipped, but they decided to stop doing that:
Before version 2.0, Elasticsearch was provided as a JAR with some (but not all) common dependencies shaded and packaged within the same artifact. This helped Java users who embed Elasticsearch in their own applications to avoid version conflicts of modules like Guava, Joda, Jackson, etc. Of course, there was still a list of other unshaded dependencies like Lucene that could still cause conflicts.
Unfortunately, shading is a complex and error prone process which solved problems for some people while creating problems for others. Shading makes it very difficult for developers and plugin authors to write and debug code properly because packages are renamed during the build. Finally, we used to test Elasticsearch unshaded then ship the shaded jar, and we don’t like to ship anything that we aren’t testing.
We have decided to ship Elasticsearch without shading from 2.0 onwards.
Please note they too refer to shaded dependencies, not shaded jar
Let me answer the question with the help of the software actually responsible for creating shaded jars ... at least when using maven.
Taken from the Apache Maven Shade Plugin home page:
This plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade - i.e. rename - the packages of some of the dependencies.
A shaded jar aka uber-jar aka fat jar will by default contain every dependency that is required to run the Java application so that no additional dependency is required to be in the classpath. You only need the correct Java version to run your application. A shaded jar will help to avoid deployment/classpath problems, but it will be much larger than the original application jar and it will not help you to avoid jar hell.
