3

I'm working on a application which uses event-sourcing and CQRS to define it's domain model.

Background

We have implemented projections to aggregate stream of all domain events into a read models used by the application. But I think our current implementation is quirky and finicky as we tinker with it frequently. So I was thinking if there are any alternatives of projection implementations.

Currently our architecture is SQL database with custom layer to store events and handle the projection event loading. We have two types of projections, local and global.

Current requirements

Local projections store model in memory, so each service instance has it's own copy. This also requires all events to be reloaded after each restart. And as the amount of events grows this takes longer and longer. We did some optimizations for some projections to not need whole event store, but it is not total solution. We like these projections as they are easy to implement and allow experimentation before the read model is stabilized.

Global projections/processors are shared across all service instances. Our architecture handles tracking processed events throuth locks stored in database. This way, event is processed only once. These projections can either update a specific table containing the model. Or it can run something like document generation or file upload for each event. Global projections are good because they are fast, but they take more effort to implement and are harder to change. They also take an hour to build from scratch due to loading, and processing of all events and updating database frequently during rebuilding.

There are also problems of dependencies between projections. We are beginning to realize it would be great if processing of one projection could depend on another projection. Maybe one projection sending events to another projection with those events not being domain events? Or process waiting for a projection to process specific events. Right now, our current implementation doesn't feel stable or extensible enough.

How we got here

All of our projections started as in-memory local projections. This allowed us to quickly roll out our service and made it easy and quick to fix the projections. But as amount of events increased, the startup time also increased due to all projections needing to process all the events. We solved that by either extending the projections to only load only some amount of latest events and making them robust against missing events. Or by rewriting them as persistent global projections, which was mostly straightforward but made subsequent modifications more difficult. That is because it requires changes to database schema and rebuilding of the changed global projection on startup, which takes time. It is also complicated by both versions of the projection needing to coexist as old one is still used when new one is being rebuilt.

The problem

I would be surprised if these problems aren't shared by many who implement projections in event-sourced projects, but I'm having hard time finding concrete implementation of production projections. Most I find are sample projects used to demonstrate CQRS with minimal concerns for ease of development, performance or memory consumption.

I've looked at event-store databases like Event Store, and that is useful for storing and querying of events. It does provide event streaming feature, but that is not durable and doesn't track multiple clients and what they processed.

So I've looked at message brokers like Kafka, RabbitMQ or ActiveMQ, and they do provide easy ways to deal with multiple consumers, which would be the projections and would handle both local and global projections. But they do not store all the events. This causes problems because replay of all events is something needed for both local projections and for new or updated global projections. I was thinking about implementing custom reload-all-events logic, but that feels like just different kind of complexity.

This then makes me wonder how others handle these kind of problems? Maybe there is library, framework or system that could simplify implementation of projections or processors in both local and global configurations and handle both storing of events and coordinating processing of those events across multiple running services. But I'm not aware of such a thing.

13
  • 1
    No incremental/periodic snapshots? May 23, 2021 at 0:12
  • @ShadowsInRain We found snapshots not that useful in our use cases. Either local projection can load only some latest events. Or the state is too big and projection's model needs to be persisted into table.
    – Euphoric
    May 23, 2021 at 5:48
  • Found this article, which states something which looks pretty obvious to me "in memory projections [..] work well for prototyping, when the number of events is fairly small". For any real system, each projection must be stored/persisted somewhere (at least snapshot-wise), otherwise you are doing Shlemiel's painting algorithm
    – Doc Brown
    May 23, 2021 at 7:27
  • @DocBrown Yeah. We expect most of our projections to become persisten ones. But that doesn't remove need for full eventstore replay for in-memory projections used in debugging and prototyping and new/updated persistent projections.
    – Euphoric
    May 23, 2021 at 7:29
  • 1
    So you're rather looking for a vocabulary/workflow to express projections tersely? After re-reading the question few times, I still have an impression that performance is the crux. Also, the long wind-up time of a persisted projection tells me that it wasn't implemented incrementally. Or maybe that's because you're excluding the projection from the deployment data. (See, I am being distracted by the performance again. Could you edit the question to make it more focused?) May 23, 2021 at 9:15

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.