I'm experimenting with event sourcing for an application we haven't build yet. No, I won't implement this without any thought, I'm just experimenting.

My domain model looks somewhat like this. The whole model is created at once via a REST service call by external parties. Please note that the status history entities are not event-sourcing related. These come from some workflow engine we are using.

Domain model invoices

After that point, there are smaller changes to the model. some details are changed. Some status history is added. Sometimes Invoice items are added. Several applications change different aspects to this model, but not as invasive as the original creation of the model. Also there are several application that need to know details about the model, but in different forms. This is what led me to believe event-sourcing could be a good fit.

My main question: Should the first creation of my model be one big event that says "Invoice bundle created" including all nested details needed for creation (in JSON or other form). Or should i break up this initial event in several smaller events, somewhat like the picture below.

Events creating invoice bundle

My second question: how should I deal with the files delivered? Should I store those as blob storage and refer to them from events or should I include those as base64 string in my events?

  • 1
    I recommend you to use small events. Granularity of information is better. And, if a problem occurs, you will be able to track at which event occurs the problem. Second question: its depend the purpose of the storage? delivering to customer?archive for long duration? – mik3fly-4steri5k Jan 24 at 17:38
  • 1
    In general, if the blobs are not changing, it's probably better/more efficient to have them saved somewhere outside the event and reference them for when you actually need to use them. – Berin Loritsch Jan 24 at 17:38
  • @BerinLoritsch only downside would be that all microservices that need access to the files also need access to the same database where those files are stored. Sharing a database between microservices is considered an anti-pattern, but I imagine sharing never changing files in a file storage could be an exception. – annemartijn Jan 25 at 10:37
  • @mik3fly-4steri5k There's one application that actually accesses these files. Beside that, it's only for archiving for long duration. I agree smaller events are easier for tracking problems. So this initial action for creating the whole tree of entities would provide a large streams of events, this isn't considered bad practice? – annemartijn Jan 25 at 10:41
  • This isn't considered bad practice if stored event are useful. If you store useless information you drown the fish. – mik3fly-4steri5k Jan 25 at 14:35

I think the answer to your question comes from defining what the failure states for your application are. Event based systems come in a couple flavors: persistent queues and objects passed in memory. Both of those have different types of failure states. For example, persistent queues write the event to disk until a client confirms receipt. Objects passed in memory no longer exist if the process dies.

Impacts of large messages:

  • Potentially can fill up a small drive with persistent queues
  • Increases the memory footprint in your process
  • Impact of a lost message includes all the content of the message

It's more efficient to store data properly and simply reference it in your messages. That helps minimize impact to your persistent queue and the memory footprint your application has. Both of those mean you can run your application with less memory, and there is less chance for your event delivery system to get clogged.

As to whether you use one message or several depends on what your consistency needs are, how atomic you need actions to be, and other similar factors. It's easier to debug problems and ensure all updates are complete when you have fewer messages to disseminate and consume. That said, not every subsystem needs the same amount of information.

You'll have to figure out what the appropriate balance is between message size (one individual message) and message volume (all the messages that make up an event).

  • Don't embed files, put them in blob storage and reference them
  • Think through failure states, and failure impacts
  • Try to do as much as possible with the messages you use (i.e. reference data instead of embed it)
  • Try to think about the impacts in terms of system resources, and debugging
  • Identify the guarantees you need
    • Do you need persistence?
    • Do you need dead letter queues?
    • Are there messages that are OK to drop?
| improve this answer | |

It really depends on what the information is for, if the intermediate state changes need to be preserved. For example, the analytics team might care in which order items were added to the cart, at what time (and which parts of the web site were visited leading to additional purchases). They probably want to send a reminder to the user if the check-out was never completed. They probably want a fast/microbatching framework that can handle a lot of events.

Whereas order fulfillment couldn't care less how the cart came to be. In fact they don't want to know about it if the cart was abandoned. They probably want a reliable framework with execute-once guarantees.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.