2

I am working with a small team that is developing a CQRS/ES "semi-microservice architecture. We are pretty far along, but running into some interesting challenges with our projections and further challenges are we start to move our projections out into a reporting database to handle cross domain concerns. I realize these are complex problems and there's no one size fits all solution. It's my first time using a heavily event based architecture so please forgive me if I am using some of the wrong terms. Perhaps this is why I am having a hard time finding further information to tackle these challenges. I am not expecting anyone to solve these problems for me, but I would be very grateful for help with terminology and if you could point me to any resources that may be helpful with the problems I will outline below. Thanks in advance!

Alright so my team and I are building software which has several services. Each service uses a CQRS architecture, eventing and some entities are event sourced. Entities that are event sourced are event sourced because there are dependencies on specific versions of those entities. Our domain event architecture is heavily inspired by Vaughn Vernon's "Implementing Domain-Driven Design". Each service has it's own relational database or at least it's own schema which is treated as a separate database. Each database has a domain event table with a constraint on the aggregate and version number to ensure a transaction fails if two events come in at the same time for the same entity. This is a heavily collaborative application so this is very important to us.

Problem #1:

We are currently publishing our domain events to subscribers. The subscribers are currently limited to the service itself and are usually projections. We tried publishing the event after a transaction is processed successfully, so the events could be processed asynchronously without holding up the user, but this led to events being processed out of order. We now process most events inside the transaction. This works for now because our projection handling logic happens very quickly, but this may not be the case for very long. The projections don't have to be updated in real-time, but they do need to be updated in near real time. We can probably allow for delays up to 2 or 3 seconds. How is event order typically guaranteed in this scenario?

Problem #2:

We are beginning to require complex sorting and filtering on views that combine data from several services which seems to necessitate that we move our projection logic out into a separate reporting service. We've looked at a few different models such as push based mechanisms or pull based mechanisms inspired by Kafka, but we're having a hard time determining how we get the events out of each service and then how we can process them in order, by aggregate, in our reporting service (especially if we are running multiple instances of our reporting service). We do recognize that based on our current setup that we can only guarantee order within services and not across services, but this is acceptable as we expect these operations to be commutative in that (the aggregate of 4 events from service 1).aggregatedWith(the aggregate of 5 events from service 2) == (the aggregate of 5 events from service 2).aggregatedWith(the aggregate of 4 events from service 1). The same 2-3 second delay is also acceptable here. Any resources or search terms on this type of problem (or any alternative suggestions) would be much appreciated!

3

Have a listen to Greg Young's talk on Polyglot data; he may persuade you that you want a pull model, rather than a push model, for your subscriptions.

Essentially, when the subscription "wakes up", it refreshes its local copy of the event history/histories from the book of record, and then writes out the new projection from those histories.

If you store metadata with the projection, you can keep track of where you left off, which can reduce the amount of redundant information you fetch from the book of record (assuming it supports ranged queries).

Fundamentally, the book of record is a database, not a service (the "service" is responsible for publishing changes to the book of record). You get events out of it by sending a query. You probably won't send the queries directly to the book of record; the actual database being used is an implementation detail that you may want to change, and in any case it's probably domain agnostic. Your database (which supports domain specific queries) is a facade in front of the persistence appliance that you've chosen.

1

I would suggest looking into the Actor Model. Employing this pattern could probably solve both of your current problems, and help standardize your design and reduce future problems as the number of event consumers grows and your consumers become more diverse. Using an Actor Framework, like Akka or Akka.net, could also be very useful, but you may be too far along in your implementation to make such a significant architectural change.

Essentially, each actor is responsible for one thing (say, a specific projection), and has a dedicated mailbox (message queue) for the events it has to process. Actors process one message at a time, so standard message-queuing semantics ensure in-order processing, since each mailbox has only one consumer (the actor itself), regardless of how the mailbox is implemented (RabbitMQ, Kafka, In-Memory Queue, etc.). Once you've decided how to implement your actors, you only need to ensure that the events are written from the event journal (your domain event table) into the actors' mailboxes in the same order they are committed to the event journal. Solving this problem once (getting events from the journal into the mailboxes in-order) would solve it for all cases, since all consumers would be Actors.

  • yes I was looking at this the other day - I haven't devoted enough time to understand it entirely, but yes I think we are too far along to consider this now – Jordan Jan 3 '18 at 18:26
  • @Jordan -- I think you will continue to have problems if you don't standardize the way consumers get messages from the event journal. If you don't have it already, I would suggest at least looking at a message queuing technology or a platform like Kafka that can keep track of your which messages each Consumer has processed and handle the dispatching of messages to the consumers. – Aaron M. Eshbach Jan 3 '18 at 18:47
  • yes we are currently using Active MQ (although I'm not sure if this has the ability to keep track of messages failed to deliver to subscribers that are not durable). I think I'm leaning towards a pull-based solution as @VoiceOfUnreason suggested which I believe you can do with Kafka. I believe there is also a way of partitioning your queues so events for the aggregates you're concerned with are grouped together although my understanding of this is still a bit foggy and I don't yet understand how this would work with multiple instances of the same service on the consuming end. – Jordan Jan 3 '18 at 20:30
  • 1
    @Jordan -- Partitioning in Kafka is used to provide for parallel consumption of messages within a Consumer Group. Kafka keeps track of the Offset position for each Consumer Group per topic per partition. When a Consumer in a Group consumes a message from a partition, the offset for that partition on that topic is incremented (they can be manually committed as well, but that's off-topic for this discussion). That way, no two Consumers in the same Group will ever get the same message, but each Consumer Group will get every message on the Topic. – Aaron M. Eshbach Jan 3 '18 at 21:03
  • so what if one consume gets messages 1-5 then another consumer gets message 6 before the first is finished processing 1-5 – Jordan Jan 3 '18 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.