Why are packages and modules separate concepts in Java 9?

Question

Java 9 will have modules in addition to packages. Usually languages have one or the other. And most programmers perceive two terms as synonyms.

Modules are built on top of packages, treating them as primitives.

Composite pattern suggests to treat primitives and composites uniformly. Otherwise bad things will happen. For example, look at project Valhalla, where they try to retrofit common supertype for primitive (value) and reference types.

Do modules and packages represent semantically separate notions?

Meaning it is sensible to have both for any language (separation of concerns). Or Java has to have both as a tribute to backward compatibility?

Why introduce a new concept instead of augmenting the existing one?

---

JSR 376: "Java platform module system" implemented within project Jigsaw.

According to SOTMS

A module is a named, self-describing collection of code and data. Its code is organized as a set of packages containing types, i.e., Java classes and interfaces; its data includes resources and other kinds of static information.

JLS carefully avoids defining what is a package. From Wikipedia:

A Java package is a technique for organizing Java classes into namespaces similar to the modules of Modula, providing modular programming in Java.

I know that quoting Wikipedia is a bad practice, but it reflects common understanding. From the entry on modular programming:

The term package is sometimes used instead of module (as in Dart, Go, or Java). In other implementations, this is a distinct concept; in Python a package is a collection of modules, while in the upcoming Java 9 the introduction of the new module concept (a collection of packages with enhanced access control) is planned.

Answer 1

The concept of a module is different from the instantiation of that concept.

Java has always had modules. A method is a module, so is a class and so is a package. A module is a unit of organisation in which internal details are hidden, and that communicates with other modules via agreed contracts. For example, a method is a module because it has hidden internals (the code and local variables) and a contract (the parameters and return type). Modules can be composed out of lower-level modules, e.g. classes contain methods.

What's missing in core Java (pre-9) is a deployable module. All the above kinds of module are not deployable units that can be copied around. Java does have a deployable artifact called a JAR file, but these are not modules because they have no encapsulation or contract: at runtime JAR files disappear, all merging together into a single "classpath".

OSGi addressed the lack of deployable modules in 1998 with the concept of a "bundle". These are physically JAR files and they contain packages, but OSGi defines additional metadata along with a runtime system to support encapsulation and contracts at that level.

Java 9 addresses the lack of deployable modules in a similar way to OSGi. Arguably this was completely unnecessary because OSGi exists and works, but that's a whole different discussion...

Unfortunately Java 9 muddies the waters by naming the new module concept just a "module". This does not mean that methods, classes and packages stop being modules! A J9 "module" is just another instantiation of the module concept. Like OSGi bundles, J9 modules are made out of packages and they are physical artifacts (usually JAR files again) that can be copied around. The runtime system understands and reifies them.

Summary: yes J9 modules and packages are semantically separate notions. Obviously Java has to retain its existing package concept for backwards compatibility. Note that the word "package" is used quite differently in Java than in other languages or in package management systems like RPM. The new J9 modules (and OSGi bundles) are much more like packages in RPM than Java packages ever were.

Answer 2

Let me hazard at an answer, though much of it may be assumptions/splitting hairs/ranting, etc.

Are they the same thing? Well, yes ^_and no

From this JavaWorld article about "Modularity in Java 9":

Packages as a modular solution

Packages attempt to add a level of abstraction to the Java programming landscape. They provide facilities for unique coding namespaces and configuration contexts. Sadly, though, package conventions are easily circumvented, frequently leading to an environment of dangerous compile-time couplings.

As @user2418306 (OP) hinted, "modules are packages done right". Modules and packages in Java (as of Java 9, obviously) are (as OP asks) semantically the same thing. That is, they are collections of pre-compiled JVM bytecode, with other metadata - essentially, they are libraries.

Well what's the difference?

However, the difference is in the metadata inside each of them. Java package manifests, or JAR manifests, aren't often maintained by library developers, nor do they provide any sure contract as to what the JAR/package provides. As explained here (JavaWorld article again):

Aren't JAR files modular enough?

JAR files and the deployment environment in which they operate greatly improve on the many legacy deployment conventions otherwise available. But JAR files have no intrinsic uniqueness, apart from a rarely used version number, which is hidden in a .jar manifest. The JAR file and the optional manifest are not used as modularity conventions within the Java runtime environment. So the package names of classes in the file and their participation in a classpath are the only parts of the JAR structure that lend modularity to the runtime environment.

---

A rant about other languages/environments, etc

Another area discussed in that article are systems such as Maven, which manage dependencies for you as part of the build process. From this page on the Apache Maven site:

Dependency management:

Maven encourages the use of a central repository of JARs and other dependencies. Maven comes with a mechanism that your project's clients can use to download any JARs required for building your project from a central JAR repository much like Perl's CPAN. This allows users of Maven to reuse JARs across projects and encourages communication between projects to ensure that backward compatibility issues are dealt with.

Now to talk about the future as if I've been there

As that page mentioned, other languages (such as Perl), have package repositories (like CPAN). This is an increasing trend (I say because I feel like it, with no emphatic proof whatsoever), over the last ~decade or so. Tools such as Ruby's gems, Python's PyPi, and the Node package manager (npm) build upon this to provide a consistent way of configuring an environment (either development, build, test, or runtime, etc) with the right stuff (packages, modules, gems, gizmo's, etc). An idea (I feel) was "borrowed" from Linux® distribution systems such as Debian's apt, RedHat's rpm, etc. (Although, obviously, has evolved at least one generation, and made these things all nicer.)

---

Java modules, although they don't necessarily add anything you cannot do already, make tooling for dependencies/package management and automated build environments all MUCH easier. Whether that makes modules "better", I decline to say. :P