Design solution for communicating between multiple layers

Question

We are developing a framework that has several layers and would be deployed in a multi-threaded environment. Each layer may have its own input/output data type. The top layer takes the input, performs some actions on the data and passes it to the next layer. Based on the output from next layer, the top layer would perform another set of operations on the data. Similar interactions happen between other layers.

---

There are two use cases that we did not incorporate in the original design:

1. Generating metrics over what operations each layer performs on the data.

2. The change in data in one layer affects the operations in another layer, the changes should be recorded and each layer should have capability to fetch the list of changes.

After some thinking we came up with two approaches to accommodate these requirements in the existing design:

1. Use observer pattern and let each layer report its changes. These changes can be then pulled by the layer that needs to use it.

   Pros:

   • Since we have already come up with a design for the framework and most of work is complete. This approach would not require significant changes to the existing design, if any.

   Cons:

   • It becomes difficult to manage(record and report back) changes corresponding to each layer.
   • There is a central class that aggregates all the data for each request. This class acts like a global variable. It has to be initialized and deleted by the top layer. Adding another top layer could be error prone.

2. Extend the current data classes using an interface equivalent to EventContainer. This way all the layers would record their changes and spit them in their output.

   Pros:

   • This approach is much more cleaner and extendable than observer pattern.

   Cons:

   • It would require significant design changes.
   • The concept of data having events as well merges two separate concerns into one and does not seem like a good idea.

---

We would like to know if there is a design pattern or any other solution that solves this problem?

Which of the above two solutions should be given higher preference considering that we would want a flexible, extendable and cleaner solution?

Answer 1

We would like to know if there is a design pattern or any other solution that solves this problem?

To me this sound like the kind of problem that Enterprise Integration Patterns solve by using communication channels and message passing.

The patterns can be use to decouple the communication between applications, between layers or between components.

There is a number of frameworks that can be used to implement this type of applications like Spring Integration, Apache Camel and Mule

What you describe sounds to me like a Service Activator. One of many different types of message endpoints (ME). The communication between your components will happen through message channels.

The components/service activators are totally unaware of each other and the channels define the pipeline that carries the messages between them.

Most frameworks, like Spring Integration, provide you with interceptors to get a chance to do something with the messages as they flow through the pipeline (equivalent to your observer pattern).

Interesting metrics can also be captured in the form of MXBeans as the message flow through the pipeline.

+----+ channel +----+ channel +----+ | |---------------| |---------------| | | ME | <-message-> | ME | <-message-> | ME | | |---------------| |---------------| | *----+ +----+ +----+

In this type of frameworks the channel abstraction gives you a lot of power. The channel could be direct channels or publish subscribe channels, and the hand off of message can be synchronous or asynchronous. So you have total control of the message flow, the span of transactions, etc.

Answer 2

I have no concrete suggestions, but I do have some questions which might help clarify the situation.

---

What needs to be captured?

• Just the final (current) data state; i.e. just about the results, not about the processes.
• Snapshots of the data states at various time points.
• An abridged description of the processes that were applied.
  "abridged" as in: trimmed-down, summarized; as opposed to the "perfect capture" described next.
• A perfect capture of the parameters of every single process that has ever happened;
  "perfect" as in enabling a perfect "record-replay" of all data transformations at a later time;
  as in data provenance in scientific data management.

---

What is the overall paradigm of the system?

This question is not about the programming languages or coding style; it asks about the manner in which data changes are propagated throughout the system.

• Imperative / command-and-control
• Event-based / rule-based / triggered / logic / reactive
• Pull-based / functional / Lazy-evaluation / on-demand
• Goal-based / optimizational / algorithmic

Imperative

Sample description: An upper-echelon (boss) class sends one or more processing commands to a number of lower-echelon (subordinate) classes. Each lower-echelon class breaks down the command it received, and sends those sub-commands to some lower-level peons (workers) for processing.

Event-based

Sample description: A number of triggering conditions are registered: "if THIS happens then do THAT". Whenever the data is changed, the triggering conditions are checked; if there are any matches, the registered actions are performed. After the registered actions are performed, more triggers could be activated, resulting in a chain-reaction.

Pull-based

In the beginning, all that is defined is a set of placeholders (identifiers) for every piece of data, along with black boxes that transform some pieces of data into some other pieces of data.

The final (ultimate) consumer of a piece of data makes the request to the system. The system checks to see if that data is available; if it is not, it searches for the black boxes that would be able to produce that data. Once one is found, it looks at whether the black box has all of the prerequisite data to start running. If it doesn't, the system tries to search again for upstream black boxes that would prepare those missing prerequisites. Ultimately, the search would hit some pre-existing data in the datastore, and therefore the processing pipeline can be kick-started.

Goal-based

The user of the system specifies the desired goal-state or optimality metric, such as cost-functions to be minimized for, or other performance metrics to be measured on the data. The WHAT is specified; the HOW is not specified. That is, how the system reaches that goal (or, whether that goal is reachable or not) is not specified by the user. The system is supposed to solve that mystery somehow.

Such systems make heavy use of research in computer-science, algorithm and mathematical theory. These systems are highly specialized in the type of solutions they can search for.

Due to the nature of such systems, the current overall state is usually in plain view by every component inside the system. A wide variety of approaches (paradigms) are used in the implementation; all of the aforementioned paradigms (imperative; event-based; pull-based; algorithmic; iterative) can be used in combination to achieve the goal.

Answer 3

Making the whole communication between your layers event-based would probably solve your problems. But instead of adding the event mechanics to the existing paths of communication (which might merge two separate concerns, as you wrote), you could replace the existing way of communication completely by an event-based system.

If each of your layers communicates with each other layers just by events, it will typically be only a small step to make those layers independent from each other, which will help to decrease the overall compile-time coupling in your system, with advantages like easier testing, easier monitoring, easier event-recordind.

Two architectural patterns which can help you to achieve this are Command Query Responsibility Segregation (CQRS) and Event Sourcing. Especially the latter will persist every interesting event, which will make it very easy to solve exactly the two problems you mentioned at the beginning of your question.

Of course, as you have correctly noted by yourself, this might be a significant design change, and you have to decide by yourself if that is feasible in your current situation.