I'm working on a application which uses event-sourcing and CQRS to define it's domain model.
We have implemented projections to aggregate stream of all domain events into a read models used by the application. But I think our current implementation is quirky and finicky as we tinker with it frequently. So I was thinking if there are any alternatives of projection implementations.
Currently our architecture is SQL database with custom layer to store events and handle the projection event loading. We have two types of projections, local and global.
Local projections store model in memory, so each service instance has it's own copy. This also requires all events to be reloaded after each restart. And as the amount of events grows this takes longer and longer. We did some optimizations for some projections to not need whole event store, but it is not total solution. We like these projections as they are easy to implement and allow experimentation before the read model is stabilized.
Global projections/processors are shared across all service instances. Our architecture handles tracking processed events throuth locks stored in database. This way, event is processed only once. These projections can either update a specific table containing the model. Or it can run something like document generation or file upload for each event. Global projections are good because they are fast, but they take more effort to implement and are harder to change. They also take an hour to build from scratch due to loading, and processing of all events and updating database frequently during rebuilding.
There are also problems of dependencies between projections. We are beginning to realize it would be great if processing of one projection could depend on another projection. Maybe one projection sending events to another projection with those events not being domain events? Or process waiting for a projection to process specific events. Right now, our current implementation doesn't feel stable or extensible enough.
How we got here
All of our projections started as in-memory local projections. This allowed us to quickly roll out our service and made it easy and quick to fix the projections. But as amount of events increased, the startup time also increased due to all projections needing to process all the events. We solved that by either extending the projections to only load only some amount of latest events and making them robust against missing events. Or by rewriting them as persistent global projections, which was mostly straightforward but made subsequent modifications more difficult. That is because it requires changes to database schema and rebuilding of the changed global projection on startup, which takes time. It is also complicated by both versions of the projection needing to coexist as old one is still used when new one is being rebuilt.
I would be surprised if these problems aren't shared by many who implement projections in event-sourced projects, but I'm having hard time finding concrete implementation of production projections. Most I find are sample projects used to demonstrate CQRS with minimal concerns for ease of development, performance or memory consumption.
I've looked at event-store databases like Event Store, and that is useful for storing and querying of events. It does provide event streaming feature, but that is not durable and doesn't track multiple clients and what they processed.
So I've looked at message brokers like Kafka, RabbitMQ or ActiveMQ, and they do provide easy ways to deal with multiple consumers, which would be the projections and would handle both local and global projections. But they do not store all the events. This causes problems because replay of all events is something needed for both local projections and for new or updated global projections. I was thinking about implementing custom reload-all-events logic, but that feels like just different kind of complexity.
This then makes me wonder how others handle these kind of problems? Maybe there is library, framework or system that could simplify implementation of projections or processors in both local and global configurations and handle both storing of events and coordinating processing of those events across multiple running services. But I'm not aware of such a thing.