3

Thinking within a Service Oriented Architecture / Microservices / Enterprise Integration framework how does one avoid infinite loops when publishing messages between systems, especially when one has limited control over certain services (e.g third party SaaS products).

For example:

  • System A tracks Entity E – I have very little control over how System A behaves, it is propertiary.
  • System B also wants to track Entity E.

Problem:

A --> B: E is created, A publishes a message to B that E has been created.
B --> A: B issues a new global identifier for E and sends this back to A.
A --> B: A sends a new message about this latest update to B (it is not aware that the update has come from B).

The problem is that A will now recieve a new update and publish a new message to B, which may in turn send a message back, starting an infinite loop / cycle.

Because I have little control over the internal behaviour of A I must handle this problem within system B.

How can we avoid these kind of problems?

I have considered

  • etag style hash key against entity E which system B can compare to what it knows of E to avoid publishing further messages on changes
  • comparing udpate timestamps, this will not work because messages from B can cause updates
10
  • 3
    Why does a message about an update of E trigger B to create a new global identifier? Please clarify. – Doc Brown Aug 31 '20 at 11:46
  • What information is included in messages published by A? If you can detect that A said "the global id was changed to X" and B says "Oh, but the global Id is already X" then B can ignore the message? – Greg Burghardt Aug 31 '20 at 16:19
  • There is a question about the service boundaries. That is why Doc Brown has asked, why does one microservice do the work of another microservice? – null Aug 31 '20 at 19:02
  • @DocBrown in this case service B is interested in synchronizing information about E into a set of other systems, C, D etc. A, C, D are all SaaS products which I cannot control the behaviour of. B coordinates this synchronization. The originating system does not have to be aware of the global identifier per se but there are other cases where similar cycles could emerge. – Max Sep 1 '20 at 10:11
  • @GregBurghardt either within the message itself or with other rpc calls B can determine the entire state of entity E, so yes it could possibly avoid handling messages it thinks are a result of such cycles. My idea for this would be to hash E and compare to a previously seen hash, but I doubt this guarantees against cycles and could result in oscillation – Max Sep 1 '20 at 10:14
2

In a comment, you wrote

For example A might manage a customer's bank information but C manages their credit status, all syncs are one way depending on which set of information we're handling, with each field having a defined source of truth.

So the actual problem here is, changes to some attributes of an entity are initiated in one system, and "mirrored" attributes exist in another system. Changes can occur from both sides to certain attributes, which cause "change events" in both directions, maybe causing infinite loops.

One way to break the event cycles is to issue events only when attributes really change their value. To stick with your example, the "global identifier" seems to be a mirrored attribute:

A --> B: E is created, A publishes a message to B that E has been created.
B --> A: B issues a new global identifier for E and sends this back to A.
A --> B: A sends a new message about this latest update to B (it is not aware that the update has come from B).

... and now B reacts to the event, informs itself about the actual changes in E, compares attributes like the "global identifier" to the current values in it's mirror of E, and observes there is no real change. The "global identifier in E stored in A" is the same "global identifier in E stored in B". And since their has nothing really changed, there is no reason to issue any further events.

If B works only as a "event mediator" to another system C, then C has to decide if a "change event" from another system really causes a data change, but the principle stays the same: compare the incoming attribute values to the current ones, and if values haven't changed, do not generate any further event.

This will work even if changes to the same attribute can be caused in different systems, without having to define a "single source of truth". The latter, however, can help to make the system behave more predictable and stable.

2
  • Thanks, I think you've summed it up better than I could. – Max Sep 3 '20 at 14:49
  • This is what I've gone with for now. For each entity we're tracking we're identifying a set of fields we're interested in tracking / syncing across systems and building a sha1 of their contents when receiving change events, which we then can compare to a cached hash from last time we saw the entity. No change, no further message propogation. Leads to 3 calls when something is first created but at least it then halts. – Max Sep 3 '20 at 14:52
0

That's awfully bad design.

Either explicitly or not, but essentially you're dealing with transaction, that covers some parts of the distributed system are in to.

Transaction has it's own unique identifier.

Transaction has it's own state machine, at least: in_progress, commited, rolledback. Let your business logic to dictate what that state machine it has, yet it always exists.

All involved activities should go under the same transaction. If your system is sophisticated that much, you might need to build and maintain a tree of transaction, where parent transaction may create child subtransactions and track their progress. Whatever. Before you go ahead and do something, you check the latest status of your transaction and make sure that the following change is still valid and needed. Otherwise you simply skip it.

Not let's step out.

Microservices topology is nothing but a directed graph. Which may contain cycles as you mentioned. How do you traverse directed graph avoiding infinite loops? You track the nodes you've seen since traversal start. The approach I've described above is the same idea projected to your problem.

2
  • Thanks for your answer. I probably shouldn't have put microservices in the title, SOA would be applicable. I have very limited control over A, it is a SaaS product that produces messages. I can't put any distributed transaction management into it. Thinking in terms of the graph is conceptually helpful I cannot stop any system except B generating events when something changes, even if that change originated from B, my ability to change state in A is limited to a rest api with no metadata that can influence events resulting from a post / put. – Max Sep 1 '20 at 10:48
  • @Max then you have to double-check that: 1) there is no way to inject some headers/key-value pairs that gonna be passed over by 3rd-party services as is; if there is such an opportunity, you can store your transaction there (actually, it's a common way); 2) in addition to hashing (which in some rare case will result to deny to handle absolutely valid request due to collision) you can either try to come up with more reasonable equality (domain name? list of expected IP-addresses?) or just keep so called "change log" with meta-description of recent changes to your entity and avoid double-work. – Sereja Bogolubov Sep 1 '20 at 13:00
0

In the age of SOA before the concept of microservices was born there where strict rules over the interaction between services. Atomic service calls had to be overseen by composite services thus ensuring that the case you mention didn't happen. Atomic services were allowed to call directly only technical services, like audit services, that performed a low level task without returning any new information/data.

The same holds if instead of talking about endless loops you talk about circular dependencies. Obviously if a system is very complex circular dependencies could happen also following the SOA guidelines, but in a microservices environment it is a lot more likely to happen.

To address the issue you describe a half step back might be the solution. But it would involve the role of architects supervising the work of different groups. All the microservices should be classified in specific categories, then some rules should be defined to constrain the dependencies only in certain directions between different categories and even some rules stating how services within the same category are allowed to interact. The main problem with this solution is the one that plagued big enterprises since the beginning of IT. How to enforce such rules across different groups and different company departments? It is more a political than technical issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.