Why does storm not supply a mechanism for supplying topology necessary dependent jars other than the fat jar?

Question

Following is a question that I had posted at the incubator-storm-user mailing list (verbatim). I had decided to open up the question here as well, because it also contains a conceptual side to it, which might be answered by non-storm users.

I am no python expert and am also a newbie to storm.

I have gone over the storm source code in order to see how to add jars to the classpath. Obviously, the preferred mechanism is the fat jar (as specified in Micheal Noll's tutorial and another post on the storm-user mailing list). The second seemingly available mechanism is via the USER_CONF_DIR which amounts to os.path.expanduser("~/.storm") but this does not allow for topology independent versioning of the same jars (e.g. apache-commons-aaa version x and version y). There does not seem to be a third way. Many other Java based technologies do give a way to amend the classpath. Why doesn't storm?

To generify the question a bit, why would a (clustered) java software framework/container not supply a mechanism for adding app specific dependent jars, but force you to use fat jars?

Answer 1

First off, let's talk about what Storm is supposed to be. From their front page we get:

Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!

And from their wiki we also get:

Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.
However, realtime data processing at massive scale is becoming more and more of a requirement for businesses. The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem.
Storm fills that hole.

With the "money quote" being this part:

Storm provides a set of general primitives for doing realtime computation. Storm is simple...

---

So Storm started out as something that was intended to be small, crazy-fast¹, and simple.

And while I haven't pulled the jar file to see its actual size, my expectation based upon the description of the project is that the "fat jar file" is reasonably small. If it isn't, then I think the project has missed one of its original goals.

The fact that they are attempting to create a set of primitives for this type of calculation also implies that they are trying to keep things concise. Or said another way, they are trying to avoid bloat and are focusing on providing just a core set of features.

We also need to consider simplicity being one of the primary goals of the project. Really Good Simplicity (tm) extends beyond the API calls and into the package itself. Not having a lot of configuration options makes it easier and simpler to deploy the project.

When you roll those three elements together, I think the converse of your question is an appropriate answer. Why would you want the additional configuration ability that you're describing? It will complicate deployment (therefore less simple), encourages moving away from a tight feature set, and you're potentially worrying about a trivial file size anyway.

And that's your architectural answer. It wasn't done because it was never intended to be supported that way. Architecture and design involves cutting off pathways so the shape can be formed or defined. While I certainly didn't design the Storm package, I can see where their architects may have rejected what you're asking about in order to achieve the other goals of the project.

¹ _{Okay fine, real time systems are better defined as systems that will guarantee a response within a fixed period of time.}