Communicating bulk data among microservices

Question

Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication.

My use case is to import data from upstream feed files(say 50k records) to our distributed system. So those records would end up in multiple services. Each record represents an instance of an entity in our system. Ex. a user.

We use async communication using RabbitMQ for other use cases where it is single instance of an entity.

We were thinking of batching the records, say 1000 records per message. So that we have manageable volume of messages in Queue and also make use of bulk inserts/updates within the consuming service.

What if a batch of 1000 records had partial success in one of the service.

• Because the other services might have already been updated, this will be an inconsistent state of the overall system.
• Although the 1000 records would be rejected in error'ing service due to DB transaction failure, the message will be retried until the root of the issue is fixed. Feel like this would clog the service.

Making REST calls is an option, but it's not as scalable as async communication.

Anyone came across similar use case before, if so how did you handle it?

Answer 1

I work on a system that has similar requirements - partners drop files that contain anywhere from 10 - 1,000,000 records to be processed. Some partners drop the files once a day, some once an hour, and others are ad-hoc.

Our system uses a data integration tool at its edge to validate the file, breakup into (up to) 1000 record batches, transform each batch into a normalized JSON payload (different partners can have different data formats), and send to REST endpoint of an orchestration service. Internally, the orchestration service is multi-threaded so that batches are processed in parallel. The orchestration service returns a response that includes a status for each record in the batch, and the data integration tool is smart enough to, for each record, record success, schedule the record for retry, or route the record to a terminal failure bucket.

All of the critical services used by the orchestration service are synchronous, but there is some use of messaging for non-critical processing. This approach works well for us.

Addressing some of your concerns:

First, you say REST is not as scalable as messaging. I think you're alluding to the fact that HTTP creates a connection per request, whereas message brokers typically keep connections open to avoid the setup/teardown overhead. That is the reason we send the records in batches - to amortize that overhead over many records. If you are going to use messaging, there may not be enough benefit to batching to warrant the extra complexity of processing batches.

You mention that taking an all or nothing approach to batch success/failure could clog the system. We agreed, that is why we designed our system so that "good" and "terminal failure" records are only processed once - only records that suffered some form of transient error are retried. Arguably, this logic is easier to implement with all of the critical processing being synchronous, rather than trying to keep track of all of the "in-flight" records being processed asynchronously.

Finally, you mention that records that are retried my have issues because some services have been updated while the records were waiting to be retired. That is an entirely separate issue. If your system has the possibility of that happening, then you must version your services so that an update allows records partially processed by an old version of the overall process can complete using the correct versions of the remaining services. Only after all of the data has flowed through the system and old version is no longer needed is it removed. We have a versioning policy that prohibits making breaking changes to an API, if breaking changes are required, a new version of the API is created and that old version is only removed after we know that all clients have been updated to the new version.

Hope this helps...

Answer 2

I'd not send the data, I'd send a link to the data or instructions to retrieve the data. I may be convenient to send a one-time password or batch ID with the event

i.e.

{ 
    "sync" : {
        "source" : "http://api.contoso.com/api/v1/users?batchId=36DA01F-9ABD-4D9D-80C7-02AF85C822A8"
    }
}

This way, retrieval of the data is delayed until the event processor runs, reducing the chance the data is no longer valid

It also leaves the possibility of canceling the operation at the source, for example by removing that batch ID