Service integration with large amounts of data

Question

I am trying to assess the viability of microservices/DDD for an application I am writing, for which a particular context/service needs to respond to an action completing in another context. Whilst previously I would handle this via integration events published to a message queue, I haven't had to deal with events which could contain large amounts of data

As a generic example. Let's say we have an Orders and Invoicing context. When an order is placed, an invoice needs to be generated and sent out.

With those bits of information I would raise an OrderPlaced event with the order information in, for example:

public class OrderPlacedEvent
{
    public Guid Id { get; }
    public List<OrderItem> Items { get; }
    public DateTime PlacedOn { get; }
}

from the Orders context, and the Invoicing context would consume this event to generate the required invoice. This seems fairly standard but all examples found are fairly small and don't seem to address what would happen if the order has 1000+ items in the order, and it leads me to believe that maybe integration events are only intended for small pieces of information

The 'easiest' way would be to just use an order ID and query the orders service to get the rest of the information, but this would add coupling between the two services which the approach is trying to remove.

Is my assumption that event data should be minimal correct? if it is, how would I (or even, is it possible to?) handle such a scenario where there are large pieces of data which another context/service needs to respond to, correctly?

Answer 1

There are many different expected granularities of message, depending on what it is you are trying to accomplish. Martin Fowler has a really good talk on this named The Many Meanings of Event-Driven Architecture (https://www.youtube.com/watch?v=STKCRSUsyP0&feature=youtu.be). He gives four styles:

• Event notification (spartan "something happened" messages with almost no data).
• Event carried state transfer (same as before, but with consumers reaching back to producer system for details).
• Event sourcing (smaller messages stored in their separately with a view built up around it).
• CQRS (event sourcing with the additional idea that you split update and read contracts).

I usually add a fifth type of messaging (not event driven) which is just a straight up worker queue where some process A is creating work for process B.

For the more specific scenario, 1000s of items doesn't seem unduly large. If it is a concern, though, I might consider an event sourcing system with worker queues triggered. So something like this:

[Order System] --> [Invoicing Events] --> (Invoice Order Events)
                         |                         ^
                         v                         |
                   [Invoicing Step 1]          [Invoice Order Service]
                         |
                         v
                        ...
                    [Invoicing Step n]

Each of the invoicing steps can call back to the invoicing system's microservice which wraps the event store built up from the order events.

The worker queues (all the invoicing step) messages can be fairly small with just enough data to tell the next processor what they need to get started. The messages from the order system could be small or large and as long as you've built up a system appropriate view of them it won't matter much.

I've also sketched the invoicing as fairly linear, but this doesn't have to be the case. If you have branches or sibling steps you can definitely build it out that way.