3

I have an application using Hexagonal Architecture. I want to implement a DLQ in 2 scenarios:

  1. When the event streaming adapter's incoming message is unreadable (invalid payload, etc)
  2. When the domain service identifies the event-converted-to-domain-model as an invalid model.

For #1, I am not sure how to handle this. Say I have a driving adapter, KafkaEventConsumer, that consumes incoming events. And I have a driven adapter, DatabaseDeadLetterQueue that inserts unreadable events into a table. Is it okay for adapters to interact with each other?

What do I mean by Is it okay for adapters to interact with each other?
My concern is that the domain service orchestrates usage of adapters via ports. If an adapter communicates directly with another adapter we are introducing additional coupling that would make changing those adapters difficult and would make them more difficult to test.

For #2, I could see two directions:

  1. Have the domain service call the DLQ port directly. The main problem I see with this approach is that we no longer have the raw incoming event payload, I could only dead-letter the domain object. This would also mean that anything caught in the adapter versus the domain service would have different DLQ payloads (one raw payload and one payload-converted-to-domain).
  2. Have the domain service throw an exception that is caught by the driving adapter, then forwards the invalid payload to the DLQ adapter. This has the same problem of adapters interacting with each other. It also shifts handling of the DLQ to the adapter layer so you'd have to remember to implement that logic for any new adapter implementation.

Do any of these options sound valid, or is there a more appropriate way to implement DLQs in Hexagonal Architecture?

6
  • 3
    "Is it okay for adapters to interact with each other?" --- I don't think if we can answer that question without defining "Okay". Is there a specific problem or concern with either of your proposed solutions? Commented Apr 2 at 15:20
  • The domain service orchestrates usage of adapters via ports. If an adapter communicates directly with another adapter we are introducing additional coupling that would make changing those adapters difficult and would make them more difficult to test.
    – Ryan
    Commented Apr 2 at 16:03
  • 3
    Please edit your question to respond to comments. It's too easy for this information to get lost otherwise. Commented Apr 2 at 18:17
  • Sure, edited. Thanks.
    – Ryan
    Commented Apr 3 at 1:26
  • @Ryan can you add a kafka tag to your question? It's fairly different from other messaging systems (AMQP based message queues, logical overlays like Kafka Connect...) with regard to DLQs, to the point of making your Q moot if you used these other systems. Commented Apr 5 at 8:39

2 Answers 2

5

I like explaining problems like these as human problems, because a lot of these approaches actually model how we've been defining human workers' roles. So let me rephrase your scenario:

You have a chef who cooks meals and hands them to the waiter. Sometimes, the chef cooks something inedible. Also, sometimes the chef cooks an edible meal, but the waiter knows that it's not what the customer wants.

In the first scenario, should the chef throw the inedible in the bin, or should the chef hand it to the waiter who then throws it in the bin? In the second scenario, the waiter inherently needs to have access to a bin anyway, so do we want to prevent the chef also needing to know where the bin is and how to operate it?

In the architecture of your choice, there is a strong disapproval of having multiple people depend on that bin. This isn't the only way, but it is the way that your architecture prescribes. This ensures that the chef is able to move or replace the bin without impacting the waiter, and vice versa. So the short answer here is that the waiter should be handling this.

If you go this route (see below, it's not the only option), then the communication between the chef and waiter needs to account for inedible meals. In your context, you wouldn't be passing a message to your domain, you would be passing a message parsing result (which may or may not contain a parsed message) to your domain.

Subsequently, your domain looks at it, sees if it's a success or not, and DLQs it when it isn't. It then goes on to validate the message, again potentially DLQing it when invalid. In this approach, the domain orchestrates the whole operation, and "bad message data" is seen as one of the outcomes of the domain logic.

However, there's some alternate options here. The main one which comes to mind: who says it needs to be the same bin in the first place?

Do you really want to treat bad input data, bad deserialization, and invalid (but correctly deserialized) data the same way? My guess is no, because these indicate very different problems. While it's nigh impossible to separate the first two (as these are different sides to the same coin), the validation failures are a very different kind of failure, and I don't see the benefit of lumping them into the same category.

If you follow that train of thought, then the chef's bin is a private implementation detail (and therefore a private dependency) of the kitchen alone, it's not a bin that the waiter needs to even be aware of.

What you're saying here is that bad message data is not part of the domain logic, because the domain logic only handles messages, not attempts at messages. This is not an unreasonable stance to take, and I suspect if we discuss this among a group of developers we're going to tread into the

The core issue remains one of the waiter and the chef not depending on the same bin, but it can be solved by either having one of them do all the bin work, or giving each of them their own bin.

6
  • 1
    Thanks, Flater. I like the analogy and the idea that the bin is an implementation detail. Is the 3rd option viable? The chef gives the waiter a wrong, but edible meal so the waiter hands the meal back to the chef and the chef throws it in the bin?
    – Ryan
    Commented Apr 6 at 14:17
  • @Ryan: It seems pointless. The waiter does not know how to cook food, that's the chef's job. Why would the chef had something to the waiter if he already saw something go wrong with the cooking of the food in the kitchen, only to then wait to get it back? Outside of the analogy, your domain is not knowledgeable about your deserialization process, that is squarely the infrastructure's responsibility. The domain should not be made responsible to divine if something when wrong in another layer. That would break the purpose of it being a different layer.
    – Flater
    Commented Apr 7 at 13:08
  • Hm, I missed something, maybe I have the analogy backwards. I thought the chef was the deserialization and the waiter was the domain. Is that right? Why would the chef hand something to the waiter if he already saw something go wrong with the cooking of the food in the kitchen. The chef didn't see something go wrong. Under the initial analogy the chef thinks the meal is correct: sometimes the chef cooks an edible meal, but the waiter knows that it's not what the customer wants. In the real world I could see this happen. The waiter says "this is wrong, chef" and the chef has to toss it.
    – Ryan
    Commented Apr 7 at 17:23
  • @Ryan: I misunderstood the scenario, scrap the finer points of my previous comment. In response to your actual question: it's is a bit ambiguous without distinguishing between the contract of the bin and its implementation. The domain is the one who needs to decide to bin it if it fails domain validation, which means it talks to the IBin interface and calls ThrowIn(). However, the actual implementation of that interface, which is presumable a persistent DLQ here, might be implemented by Infrastructure. Everything I said before this comment focused on the interface, not implementation.
    – Flater
    Commented Apr 7 at 23:26
  • @Ryan: Reusing the analogy, "The waiter says "this is wrong, chef" and the chef has to toss it." I would be more inclined to say that the waiter has access to a trash chute that leads to a bin in the kitchen. The point I'm making with that is that it shouldn't require a person in the kitchen (who thinks and reasons about what needs to be done), it just requires a plain old bin implementation that the waiter is able to use (via the chute, which is the equivalent of an injected interface into the domain logic here). The brain/logic is solely the waiter's, even if the bin is in the kitchen.
    – Flater
    Commented Apr 7 at 23:28
3

Though it seems different in Kafka, in AMQP based message queue implementations, the way to dead letter a message is by rejecting it (nack) through the same channel that you got it. Meaning the same adapter. I don't think anything prevents you from doing the same with Kafka. Just have your application service return the error and the same primary adapter that got the message post it back to the DLQ topic.

A good reason behind this is that the message queue is completely coupled to its DLQ mechanism, it's the same dependency from your application's perspective. It's not like sending a notification out to a third party system if something goes wrong, it's responding to the origin system with a refusal in its own communication protocol.

I'm not sure if that completely matches your option 2. but it looks like it, although I don't really see why "this has the same problem of adapters interacting with each other".

1
  • have your application service return the error and the same primary adapter that got the message post it back to the DLQ topic. Would that look something like: try: domain.doTheThing() catch exception: deadLetter(exception)? I don't really see why "this has the same problem of adapters interacting with each other". I see, the adapters are actually the same in this scenario. I was picturing them as independent adapters - one incoming to consume messages and one outgoing to publish to the DLQ. The approach you recommended works without separating them.
    – Ryan
    Commented Apr 6 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.