1

For a complex service that my team and several others maintain, two of my team's applications are called when processing one incoming request. The flow is as follows: an upstream system invokes an HTTP endpoint of our application A, then several other applications managed by other teams are called, one of them publishing an event Topic1 to a message broker that our application B then consumes. B needs to do some extra processing on this event and publish a different event Topic2 to the message broker. The part needed to serve the incoming request (synchronous) ends somewhere between A and B, and the rest is (simplifying) needed for our analytics.

The problem: in order to process the events it consumes, B needs to know some information that is known by A, and that is transient (specific to each request). In the past, we put this information in the response A produces and asked systems in between to forward it to B. But there are several systems and teams involved, which have their priorities and concerns about passing this information, taking several weeks and plenty of discussions.

In summary, we need another way of passing information from A to B, controlled only by our team. What is the best way of doing this? Something we cannot do is having B directly call A (e.g. having A locally store the info and then B fetching it). We are considering these two approaches, but can consider other solutions:

  • A publishes an event Topic3 with the needed info, B merges Topic1 and Topic3 by requestId and extracs the info from Topic3
  • A writes to a cache that B then queries by requestId
  • Are you sure you don't just need to know that all of the other sub tasks have completed? Maybe each sub task should publish an event when it is done, and the main task can listen for these events and collect all the necessary information. Sounds like this will need to turn into a very asynchronous process. – Greg Burghardt May 14 '19 at 12:44
  • @GregBurghardt We don't need only to know that other tasks have completed, we need some pieces of information. Also, I'm not sure that talking about 'main task' would be correct, I'd rather talk about most-upstream service and other downstream services. – antonro May 14 '19 at 12:55
  • 1
    "which have their priorities and concerns about passing this information" - could you edit the question to expand on that a bit? Seems like this is one of the key sources of the problem, so help us understand what is actually going on. What is the nature of these concerns? In what ways this affects them? Do they read and/or rearange this information, or simply pass it along without examining it? Maybe you just need better interfaces between the systems/services. – Filip Milovanović May 14 '19 at 13:57
  • Is it right that some teams don't want to provide that additional info when sending Topic1 events? Perhaps introducing an augmented Topic1a with events allowing to transfer the same data as Topic1 events PLUS your information specific to A is the solution? B needs to subscribe to it and perform its extended processing only on that topic's events. – Hero Wanders May 14 '19 at 18:56
2

A design allowing A to throw messages on the message bus B reads strengthens the message passing abstraction. A design requiring B reading from A's cache might be more likely to produce a close coupling of A's and B's internals.

In a message passing implementation, if Topic 3 arrives first B leaves it in its mailbox. When Topic 1 arrives, B checks its mailbox. If Topic 1 arrives first, B leaves it in its mailbox. When Topic 3 arrives, B checks its mailbox. In the generalized case, message receipt can trigger re-evaluating every message in the mailbox. This allows purging messages that are no longer relevant.

An implementation where B reads from A's cache, A will be responsible for flushing the cache. Coordinating this with the producer of Topic 1 would require further close coupling, not just with B.

0

It seems to me that what you really want is URI that provides this information on demand. The underlying implementation could be a cache, a topic (I guess), or a key-value store. Anything that allows you to quickly retrieve the information from by it the requestId will suffice. Ideally, that detail is something you want to keep out of the interaction with B.

Which storage approach you use depends on a lot of different factors. By abstracting that detail away from B, you can easily change the storage without any impact on B.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.