Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up requests as necessary but the gateway never does. What I’m finding is that the time for the gateway to respond is too long. I want to queue messages from the gateway to async the entire system and generate faster response times but I’m not sure how to guarantee persistence so we can handle failure. I’m the absence of an actual database record I’m not sure how it can handle proper persistence.

In short: how does a fully event-driven architecture guarantee persistency of data. It will be easier to elaborate with a concrete example.

We collect earning records and then later allow people to make payments based on their earnings. We have an endpoint to create an earnings record. What it looks like is the following:

client -> gateway -> earnings service

The problem with this is that it’s all synchronous. The earnings service call takes too long to do all the work. It has to write some records, do checks to make sure the data is valid, etc. There are other asynchronous portions but not this part.

If I queue at the gateway then there’s no database record yet. The first record gets written in the earnings service. In our synchronous system that’s fine because we guarantee a record on a 200-level response. If I start queueing at the gateway I now need to ensure we don’t lose messages without a db record. How would I do that?

Is there some sort of way to guarantee persistence in Rabbit? Is a database of messages an option? Our endpoints are idempotent so in a disaster recovery situation we can replay messages without harm. I’m just not sure how to go about doing this. Thanks


You want to guarantee persistence succeeded at the very moment the record was created ? Why ?

It seems more than enough for the client to know that the request was valid, and that the earnings service will take care of everything else now or later.

With that logic, you could start by validating that request in earnings, which is usually a pretty fast operation and respond instantly with a validation error if something is wrong. Then if the request is valid, store it in a queue and have another part of earnings handling these requests to create records.

Now if persistence failed, the earnings service can replay the failed events easily later on and the client knows it did its job properly which is sending the right request.

| improve this answer | |
  • Is that a fast enough process? If validation requires queries it’s not fast enough. So you want to just validate that the correct fields are there and queue the message? Does it insert a record at this point? If not how do I guarantee persistency of the queue? I want to reduce the overhead to respond as fast as possible. – Michael Yousef Apr 21 '19 at 16:41
  • You can't say a record has been inserted without it being inserted beforehand. Inserting a record into the queue should be a very fast process, way faster than inserting an earning record. And validation could be only made in memory, or could depend upon the database but again that's a sacrifice to make, it's a matter of judgement. Or there could be no validation at all, and the earnings service could simply tell it got the request. It depends again on what you need on the client side and the sacrifices you're willing to make in terms of speed. – Steve Chamaillard Apr 21 '19 at 17:18
  • Right. I’m okay with no validation and returning some asynchronous error message. Having consumers of the gateway subscribe to a callback is perfectly fine. But if I publish a message to a queue how do I guarantee that message is never lost. In the absence of a database insert that message is the entirety of our knowledge of that request. Like if the Rabbit container goes down or something is that message persisted somewhere or do we lose it? Because in the case we return before the database insert we can’t afford to lose any messages – Michael Yousef Apr 21 '19 at 17:21
  • Let's assume everything but the client is down at the moment. They can't even access the gateway. How can you not lose that message at this very moment ? There's no way. The data is on the client side and it won't go through. So you need to adapt to asynchronicity, and let the client know which records were inserted successfully (reading the database here, so this is guaranteed persistence) so they can see which records are missing so that they can retry inserting them later on. – Steve Chamaillard Apr 21 '19 at 19:26
  • If the client is down then your service isn't responding with 200-level responses. If you respond with a 200-level response you can't find the data is not persisted later on. If you respond 201 created then you have to have that data somewhere. If you're suggesting to let the client know which records were inserted by reads from the system via database then you're conceding that there should be a database insert. I'm contending that gateway -> internal service -> validation -> db insert is too much work done synchronously. – Michael Yousef Apr 21 '19 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.