Avoiding infinite loops in SOA / Enterprise Integration messaging

Question

Thinking within a Service Oriented Architecture / Microservices / Enterprise Integration framework how does one avoid infinite loops when publishing messages between systems, especially when one has limited control over certain services (e.g third party SaaS products).

For example:

• System A tracks Entity E – I have very little control over how System A behaves, it is propertiary.
• System B also wants to track Entity E.

Problem:

A --> B: E is created, A publishes a message to B that E has been created.
B --> A: B issues a new global identifier for E and sends this back to A.
A --> B: A sends a new message about this latest update to B (it is not aware that the update has come from B).

The problem is that A will now recieve a new update and publish a new message to B, which may in turn send a message back, starting an infinite loop / cycle.

Because I have little control over the internal behaviour of A I must handle this problem within system B.

How can we avoid these kind of problems?

I have considered

• etag style hash key against entity E which system B can compare to what it knows of E to avoid publishing further messages on changes
• comparing udpate timestamps, this will not work because messages from B can cause updates

Answer 1

In a comment, you wrote

For example A might manage a customer's bank information but C manages their credit status, all syncs are one way depending on which set of information we're handling, with each field having a defined source of truth.

So the actual problem here is, changes to some attributes of an entity are initiated in one system, and "mirrored" attributes exist in another system. Changes can occur from both sides to certain attributes, which cause "change events" in both directions, maybe causing infinite loops.

One way to break the event cycles is to issue events only when attributes really change their value. To stick with your example, the "global identifier" seems to be a mirrored attribute:

A --> B: E is created, A publishes a message to B that E has been created.
B --> A: B issues a new global identifier for E and sends this back to A.
A --> B: A sends a new message about this latest update to B (it is not aware that the update has come from B).

... and now B reacts to the event, informs itself about the actual changes in E, compares attributes like the "global identifier" to the current values in it's mirror of E, and observes there is no real change. The "global identifier in E stored in A" is the same "global identifier in E stored in B". And since their has nothing really changed, there is no reason to issue any further events.

If B works only as a "event mediator" to another system C, then C has to decide if a "change event" from another system really causes a data change, but the principle stays the same: compare the incoming attribute values to the current ones, and if values haven't changed, do not generate any further event.

This will work even if changes to the same attribute can be caused in different systems, without having to define a "single source of truth". The latter, however, can help to make the system behave more predictable and stable.

Answer 2

That's awfully bad design.

Either explicitly or not, but essentially you're dealing with transaction, that covers some parts of the distributed system are in to.

Transaction has it's own unique identifier.

Transaction has it's own state machine, at least: in_progress, commited, rolledback. Let your business logic to dictate what that state machine it has, yet it always exists.

All involved activities should go under the same transaction. If your system is sophisticated that much, you might need to build and maintain a tree of transaction, where parent transaction may create child subtransactions and track their progress. Whatever. Before you go ahead and do something, you check the latest status of your transaction and make sure that the following change is still valid and needed. Otherwise you simply skip it.

Not let's step out.

Microservices topology is nothing but a directed graph. Which may contain cycles as you mentioned. How do you traverse directed graph avoiding infinite loops? You track the nodes you've seen since traversal start. The approach I've described above is the same idea projected to your problem.

Answer 3

In the age of SOA before the concept of microservices was born there where strict rules over the interaction between services. Atomic service calls had to be overseen by composite services thus ensuring that the case you mention didn't happen. Atomic services were allowed to call directly only technical services, like audit services, that performed a low level task without returning any new information/data.

The same holds if instead of talking about endless loops you talk about circular dependencies. Obviously if a system is very complex circular dependencies could happen also following the SOA guidelines, but in a microservices environment it is a lot more likely to happen.

To address the issue you describe a half step back might be the solution. But it would involve the role of architects supervising the work of different groups. All the microservices should be classified in specific categories, then some rules should be defined to constrain the dependencies only in certain directions between different categories and even some rules stating how services within the same category are allowed to interact. The main problem with this solution is the one that plagued big enterprises since the beginning of IT. How to enforce such rules across different groups and different company departments? It is more a political than technical issue.