Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication.

My use case is to import data from upstream feed files(say 50k records) to our distributed system. So those records would end up in multiple services. Each record represents an instance of an entity in our system. Ex. a user.

We use async communication using RabbitMQ for other use cases where it is single instance of an entity.

We were thinking of batching the records, say 1000 records per message. So that we have manageable volume of messages in Queue and also make use of bulk inserts/updates within the consuming service.

What if a batch of 1000 records had partial success in one of the service.

  • Because the other services might have already been updated, this will be an inconsistent state of the overall system.
  • Although the 1000 records would be rejected in error'ing service due to DB transaction failure, the message will be retried until the root of the issue is fixed. Feel like this would clog the service.

Making REST calls is an option, but it's not as scalable as async communication.

Anyone came across similar use case before, if so how did you handle it?

  • Welcome to the conundrum of the CAP theorem. en.wikipedia.org/wiki/CAP_theorem If your database fails, that would cause a wider issue than just that batch of 1000 records, correct? Then if eventual consistency is acceptable you need to assess the impact of the back pressure in the queue. How likely is that scenario going to happen? It might be worth rehearsing that scenario in a test environment to get a better picture. Feb 25, 2020 at 15:58

2 Answers 2


I work on a system that has similar requirements - partners drop files that contain anywhere from 10 - 1,000,000 records to be processed. Some partners drop the files once a day, some once an hour, and others are ad-hoc.

Our system uses a data integration tool at its edge to validate the file, breakup into (up to) 1000 record batches, transform each batch into a normalized JSON payload (different partners can have different data formats), and send to REST endpoint of an orchestration service. Internally, the orchestration service is multi-threaded so that batches are processed in parallel. The orchestration service returns a response that includes a status for each record in the batch, and the data integration tool is smart enough to, for each record, record success, schedule the record for retry, or route the record to a terminal failure bucket.

All of the critical services used by the orchestration service are synchronous, but there is some use of messaging for non-critical processing. This approach works well for us.

Addressing some of your concerns:

First, you say REST is not as scalable as messaging. I think you're alluding to the fact that HTTP creates a connection per request, whereas message brokers typically keep connections open to avoid the setup/teardown overhead. That is the reason we send the records in batches - to amortize that overhead over many records. If you are going to use messaging, there may not be enough benefit to batching to warrant the extra complexity of processing batches.

You mention that taking an all or nothing approach to batch success/failure could clog the system. We agreed, that is why we designed our system so that "good" and "terminal failure" records are only processed once - only records that suffered some form of transient error are retried. Arguably, this logic is easier to implement with all of the critical processing being synchronous, rather than trying to keep track of all of the "in-flight" records being processed asynchronously.

Finally, you mention that records that are retried my have issues because some services have been updated while the records were waiting to be retired. That is an entirely separate issue. If your system has the possibility of that happening, then you must version your services so that an update allows records partially processed by an old version of the overall process can complete using the correct versions of the remaining services. Only after all of the data has flowed through the system and old version is no longer needed is it removed. We have a versioning policy that prohibits making breaking changes to an API, if breaking changes are required, a new version of the API is created and that old version is only removed after we know that all clients have been updated to the new version.

Hope this helps...

  • What you described is pretty much what we are doing using serverless functions, except that orchestration service uses async communication. But it makes sense to do sync calls for critical pieces and log errors for retry. You mentioned "terminal failure", could you give some examples in a micro-service based system? Feb 28, 2020 at 16:40
  • 1
    By "terminal failure" I simply mean any failure that is not "retryable". For example, if a service fails because it lost its connection to its database, that is probably a transient error that will go away fairly quickly (not terminal). But if the payload is missing a required field, no matter how many times you retry that record will never be processed successfully, so why even try. In this example the former failure probably returns a 500 and the latter a 400, so in this case retry 500s, but just move 400s to the failure bucket.
    – sceaj
    Feb 29, 2020 at 4:51

I'd not send the data, I'd send a link to the data or instructions to retrieve the data. I may be convenient to send a one-time password or batch ID with the event


    "sync" : {
        "source" : "http://api.contoso.com/api/v1/users?batchId=36DA01F-9ABD-4D9D-80C7-02AF85C822A8"

This way, retrieval of the data is delayed until the event processor runs, reducing the chance the data is no longer valid

It also leaves the possibility of canceling the operation at the source, for example by removing that batch ID

  • This is an interesting approach. One thing it doesn't clog the network as there is no bulk data being transferred to different services. But I guess, it adds complexity in tracking the services that had/had not consumed the data and if consumed, are all records successful, if not what are the failure ones that needs retry. Feb 28, 2020 at 16:45
  • There is no free lunch. If you want strong run exactly once promises out of your messaging service, it's unlikely to handle large data volumes (cheaply). If you have a high speed messaging service it might not offer these promises.
    – Martin K
    Feb 29, 2020 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.