Distributing Java code into packages using a clustering approach

Question

Theory is, that classes which strongly know and use each other should be in the same package (using package visibility for functions that are not used by the rest of the application), thus forming a concise, understandable interface. [Well-known classes (accessed from almost everywhere, such as loggers) should go to their own package, call it utils.]

Assume a code base of several hundred Java files, distributed rather liberally over about six dozen packages, most methods declared public (aside from some private ones), and rather obscure relations between classes. (No clear entry points either, since methods are called from a JSF infrastructure, if that matters. There may be some main() methods, but they are usually not used.)

Did someone ever think about applying a clustering approach to group such a large set of Java classes into packages? I wonder whether and how this would be possible, either to outline the strength of relations between classes as help to manual clean up, or even to automatically reorganize the code somehow.

The distance measure could involve the number of functions required to be declared public, and the number of package import statements (meaning those ending with a .*) that are necessary for the code to compile. Since the algorithmically “best” answer to that question would be to have all classes in one package (does not require any public methods or imports any more, which is not what I want), it seems to me that a hierarchical clustering approach would be the best (leaving the final granularity of packages as a decision to humans), but maybe there are better approaches I am not aware of.

I tried to find information on that topic, but searching for “Java” and “clustering” only brought me to pages explaining how to implement clustering with Java, which is not what I am looking for. (Of course, an approach may well be implemented in Java as well, since it may be realized using introspection. But that’s not important to my question.)

Answer 1

Did someone ever think about applying a clustering approach to group such a large set of Java classes into packages?

Yes, it has been a hot topic in software (re)engineering (and more specifically software restructuring) research a few years ago, with flow network cutting algorithms (for both the structural and the behavioural points of view) and some other approaches.

Some references, there actually are a lot of research papers on this topic:

• http://www.sciencedirect.com/science/article/pii/S0164121203002346
• http://www.sciencedirect.com/science/article/pii/S2212017313006671
• http://www.springer.com/us/book/9783834819536

Other notions of distance/similarity have been considered as well. For instance, modules (e.g., classes) that are often modified in the same commits can be considered as "close" to each other since they tend to coevolve over time. Therefore, determining clusters of these "similar" modules helps to prevent forgotten co-evolution needs (you changed a class, and you forgot to change that related class) and to highlight implicit (and maybe undesired) relations between modules.

Answer 2

I have actually thought about this exact topic. The idea I kicked around was to:

1. Represent the project's code base as a graph.
2. Each Class becomes a Vertex in the graph.
3. Each declared methods with Parameters becomes a directed edge. When Class A has a method that requires an input parameter of type Class B then a directed edge from Class A to to Class B is added to the graph. For example, the method (declared within ClassA) public void doWorkWith(ClassB b, ClassC c) would result in 2 directed edges being added to the graph.
4. I considered method return types and classes that were used in the implementation but not exposed at the API level. I forgot how I intended to handle these
5. Use Java reflection to create the graph programmatically.
6. Do some type of cluster and/or community detection on the resulting graph

I also found little information about this topic when I poked around online. That said, the book Agile Software Development by Robert Martin discusses this topic.

I did not implement this because: (1) Its a lot of work and (2) I would bet serious money that any project that needs to be analyzed this way would result in a horribly interwoven graph that it hard to "separate cleanly". Consequently, I focused my efforts on refactoring the project to use more Dependency Injection.

Edited to Add: One quick and easy thing to do is to merely count the number of import statements. Broadly speaking classes that begin with a zillion import statements are prime targets for improvement efforts Sadly, those are the probably the most dangerous classes to touch.

Answer 3

I disagree with your base assumption, that it is best to separate classes into packages to minimize inter-package dependencies. Following that heuristic to its extreme, we end up with all classes in the same package, which is clearly a maintenance nightmare. But even without using such a reducto ad absurdum argument, it seems to me that the most important factor in deciding where a class should be is that it logically fits into a package such that the programmers on the team will be able to guess where it should be.

Minimizing dependencies is good, but we should probably use techniques like façades or dependency inversion as the primary method of achieving this, rather them moving code so that clients and servers are together.

Answer 4

I'll propose this an answer, though it's really more of pointer to an excellent discussion and tool for visualizing these relationships, providing a starting point for refactor and exposing the effects of the refactor.

Edmund Kirwan is obsessed with this topic and his site is chock full of interesting revelations about the structure of code. His analysis tool, Spoiklin Soice, is available on his site.

Edmund's Site

The Article