6

I'm looking for information on the best approach to message validation in asynchronous-messaging-based services (i.e. services that pull messages from some sort of message queue or broker, rather than providing an HTTP-based API or something similarly request-response oriented).

In an HTTP API, if the message is invalid (wrong fields, invalid JSON/XML/Whatever) you can return a 400 error. The receiving service can go on as before, and the sender has to deal with the error. (Many HTTP libraries raise an exception when a non-200 response is returned, making the source of the problem very clear.)

With async messaging processes, I can think of a few approaches:

  1. The receiving process deals with the error somehow.
  2. The receiving service just ignores messages that are not valid.
  3. The receiving service logs the invalid message and then moves on to the next message. If things don't seem to be working correctly, hopefully, we can locate the problem in the logs.
  4. The receiving service sends an error notification back on some sort of a response channel.

None of these options seem particularly good.

Option (1) is really just a "hand-wave" in the direction of some as-yet-unknown solution.

Options (2)-(3) seem to run the risk of problems in the sender's code being masked and made difficult to identify (compared to the request-response situation, where you can immediately identify a 400 response, which occurs in a "location" very closely linked to the source of the error).

Option (4) seems to be attempting to recreate a request/response system, negating any reasons for opting for an async-messaging based system. If you're going to request/response, why would you be using a message-broker based system in the first place?

I would appreciate any information on best practices, and on how the tension described above might be resolved.

  • When you talk about 'validation' of a message, do you just mean things like "wrong fields, invalid JSON/XML/Whatever" - or do you also include more complex (basic business logic) type validation? (such as, DOB cannot be in future, references a valid order number, etc) – Tersosauros Feb 20 '18 at 0:01
  • I do mean those things, but I don't necessarily just mean those things. I mean anything which might, in a HTTP or RPC-style API, result in some sort of response saying that there's something wrong with the message. For me, a DOB that is invalid because it is in the future would probably fall into that category. – samfrances Feb 20 '18 at 0:06
  • Does the message creator(s) have any way to know if/when the message has been handled? Does your system have any way to know that message processing failed, and a message needs to be reprocessed (or abandoned)? – 1201ProgramAlarm Feb 20 '18 at 0:22
  • 1
    This links can help you in your problem – Dherik Feb 20 '18 at 0:47
1

Most* Message Oriented Middleware platforms provide a mechanism for the schematic validation (i.e. against a schema) of incoming messages as they are enqueued. This is logic that checks for things like missing/null fields, values outside of certain ranges, bad formatting, etc. But, as these checks are being done by the message broker, and not the end system - cannot check references such as customer ID exists, order exists, etc.

* "Most" of the strongly-typed message brokers, not just systems that queue dumb strings.

For example, in JMS a 'Validator' method can be assigned: https://docs.spring.io/spring/docs/4.1.1.RELEASE/spring-framework-reference/html/jms.html#jms-annotated-method-signature

I believe WCF has something similar, closest thing I could find was this article.


As for your options, I believe 4 is probably the most common. There are reasons (to do with scalability) why this response channel is preferable to a request-response driven system (I can validate your message and send you the Exception whenever, rather than tying up a socket + process/thread on each end while you wait for me to do it live, etc).

1

You can also send the message to a Dead Letter Queue

In message queueing the dead letter queue is a service implementation to store messages that meet one or more of the following criteria :

Message that is sent to a queue that does not exist.

Queue length limit exceeded.

Message length limit exceeded.

Message is rejected by another queue exchange.

Message reaches a threshold read counter number, because it is not consumed.

Sometimes this is called a "back out queue". Dead letter queue storing of these messages allows developers to look for common patterns and potential software problems.

0

I have been working with a queue based API for some years now and I've found the solution implemented to be working just fine. What they did in this API is this:

  1. Client sends a message to be handled to a REST endpoint
  2. Message is validated for correctness. If it is not valid the response is an HTTP error, 200 is returned otherwise, containing an id identifying the message in the queue
    1. If some error happens while handling the message, it is logged in a separate log file
    2. A client can always query the queue through a separate endpoint, finding out the current status of the message it queued (or querying for all errors, for example)

HTH.

0

As far as I have seen in my current project, #4 is the one used - Send the notification back to the source channel if the validation checks fail.

0

We have created a central library with the request, response objects logic for serializing, deserializing, pushing and receiving from the queue, and basic parameter validators for such scenarios.

Every service includes this common library and uses this for accessing queues. Basic parameter validations are run through this library before pushing the request and rejected on failure before pushing.

Business logic based validations are performed by the services and are pushed back on response channel/notifications are generated on failure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.