0

In our domain we have a concept of billable activities that get logged by a user. The modeling is pretty straightforward -- the User is an aggregate root, with the Activities being a collection under the user (an activity doesn't make any sense outside a user, and we have rules to enforce at that level).

We're in the process of a redesign and are evaluating a move to CQRS/ES. The modeling above makes sense, but there's another action that happens on the activities that is a challenge: those activities are "sequenced" (just assigned an auto-incrementing integer SequenceId and eventually get batched for billing, usually based on a date range or as "all activities since SequenceId X.

Due to the number of transactions (10K+ users, 20K+ activities per batch) and because we're batching based on properties intrinsic to the activity itself, it doesn't make sense to batch them through the user AR. However, the act of batching an activity does need to be recorded on the activity as it impacts the business rules for a user's actions (e.g. an activity can't be edited if it's been batched).

Now, without CQRS/ES, we just query the activities then batch flip a status property on them. That isn't possible now with the User AR managing access to those activities. This seems like a candidate for a separate bounded context, but I still am not sure the right way to share the activities between the users and the batches.

This can't be an uncommon use case, so I'm hoping someone out there has tackled it before and can offer some pointers.

7
  • whats the reason to make the activity part of the user ar?
    – Ewan
    Feb 4, 2019 at 12:47
  • With seperate bounded context, you mean a BC for Billing, right? If so, is there a reason why an Activity in the original BC should share the SequenceId with an Activity in the Billing BC? Do you even need that ID in Billing?
    – Rik D
    Feb 4, 2019 at 13:41
  • @Ewan, I added the activity as being part of the user because 1) an activity doesn't make any sense outside of a user, and 2) there's certain validation when activities are added or edited that needs to be applied, so it makes sense for the user to have all that information. I considered making it its own AR, but it makes it complicated to handle certain scenarios. For example, we don't want a user starting a new activity if there's still an open activity in progress. Feb 4, 2019 at 14:13
  • the batches are across users right?
    – Ewan
    Feb 4, 2019 at 14:15
  • 1
    I'm not convinced a User needs to own each Activity. The invariant you cite above about not starting a new Activity if there is already one open does not require User to be an aggregate over Activity. An aggregate relationship between objects is only necessary one object (User) needs to know about the state of another object (Activity) in order to enforce its invariants (changing UserName requires knowledge of an Activity). Feb 4, 2019 at 20:39

2 Answers 2

1

This can't be an uncommon use case, so I'm hoping someone out there has tackled it before and can offer some pointers.

Sometimes what you have is a hard problem.

The first issue that you have is that there is a data race in your process. Your "batch this, mark it as batched" activities are racing with your edit activities. Take a careful read through Race Conditions Don't Exist.

A microsecond difference in timing shouldn’t make a difference to core business behaviors.

This is somewhat confused by the fact that you seem to have a two phase commit thing going on (the batching process modifies something over there... and then we do book keeping on the batching process over here on the aggregate).

The second issue is part of the general category of problems called set validation. In an RDBMS, as long as all of the users are stored in the same database, you can lock the users table, perform a bulk update, and release the lock.

"Aggregates" are analogous to coarse grained locks. So what this bulk operation is telling you is that the set of all users is "the" aggregate. Ugh.

Neither of these issues are particularly related to CQRS, or event sourcing -- you can lock up all of the event streams in a single store so that you can perform "bulk" operations, but the lock is typically so painful that redesigning the system is a more cost effective choice.

One design choice that sometimes helps is to be more explicit about describing values, rather than references. "We batched activity 12345 when it was @ version 6." So when you look at the history of that activity, you might find two additional user edits version 7 and version 8, and the receipt of the batch operation at version 9, and you can then decide how much money the business wants to invest in mitigating/preventing that scenario.

1
  • The delay between batching an activity in one BC and updating its status in another could be an acceptable risk if there was a mechanism to invalidate a change made after an activity was batched. Where I'm struggling is how the batching happens. Does it persist its own type of activity based on events fired from the user? If batching is a separate BC that fires an event that a listener picks up and issues a command through the user AR to the related activity, then the batching BC will need to know the user AR version, which I guess I could push to a read model, but it doesn't feel right. Feb 4, 2019 at 14:24
1

OK so the way I see it is this:

You have a physical constraint on your process. You simply can't load all the Users with batchable transactions into memory at the same time.

This can't be a new constraint, technology isn't regressing, so your current batch process as it is implemented in the business must treat the activities as things that have meaning outside the User.

Furthermore, you can have related things that are in different ARs. For example you can choose between any of a number of options, simply by having external Ids attached to ARs rather than the full object.

You can still apply your business rules regardless of where you put the logic, by implementing states, locks or immutability in the correct place. eg

Activity.AssignToUser(string userId) // set AssignedUserId, but not status = 'started'

Activity.AssignToBatch(string batchId) //fail if status is still editable

User.AssignActivity(string activityId) //fail if User.AssignedActivityId != null

Batch.AssignActivity(string activityId) //check editable status when you load the activity to process the batch 

It's not totally clear to me how ES fits in with this. You have the same problem with a standard DB, simply calling update activity where.. without going through the AR breaks your encapsulation.

My advice would be to have the activity generate an immutable event on completion. This CompletedActivity can then be fired off to be batched whilst keeping Activity as part of the User AR.

This keeps your User related business rules in place, but also encapsulates the idea of the 'offline' batch nature of the billing process.

It also allows you scope for editing the Activity post batching, where you would presumably need to deal with amending the overall account, maybe with an AmendedActivity event.

7
  • Part of the reason we're looking at ES is that there is business value in keeping the history of events on these aggregates. Another part is that we have a lot of both read and write transactions and have had performance issues with SQL server. It's appealing to have the ability to leverage CQRS and ES to construct distributed systems with high-performance append-only event writing separated from high-performance Redis+SQL read models. Feb 4, 2019 at 15:02
  • Also, separating an activity into its own AR means we'd have to deal with eventual consistency, so we'd have to have a process monitoring for possible business rule violations that would fire commands to address them. With the activities all accessed via the user that data is always consistent and we can have a simpler design. Of course, that may come at the cost of complexity on the batching side, so maybe overall it's not as simple as it seems. Feb 4, 2019 at 15:14
  • does my final advice not neatly solve your problem?
    – Ewan
    Feb 4, 2019 at 15:34
  • It does make a lot of sense, and we thought about that concept of binding the activity to the user and the batch. The challenge that design presents, though, is that due to eventual consistency it's possible that a user could log an activity, then quickly log another invalid activity, and the system would accept the invalid activity. I was hoping to avoid having to have a separate process that would monitor for those inconsistent states and take action to remedy them. Feb 4, 2019 at 17:25
  • no no. the activity is part of the user ar. the CompletedActivity is a separate object which you send off for batch processing
    – Ewan
    Feb 4, 2019 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.