How do I identify ways to break apart an Aggregate that has become my entire application?

Question

I am developing an algorithmic trading framework using DDD/event-driven principles. I've decided the following components sit within my bounded context:

Data feed -> [Entry Point] -> "HandleData" command -> [Alpha Model] -> Insights -> [Portfolio Model] -> Targets -> [Execution Model] -> Orders -> [BrokerGateway]

Each model is a stateful (entity-like) computation. Other concerns (like performance reporting) can sit in another bounded context. Because I believe want transactional consistency throughout the pipeline (per data point ingested), I have created a single aggregate ("Algorithm"), which is composed of the above models:

Data feed -> [Entry Point] -> "HandleData" command -> [AlgorithmicTrader]

But this means that a single AlgorithmicTrader aggregate now essentially encompasses my entire application (excluding the data feed and entry point).

I'm having a hard time splitting up my Algorithm aggregate for the following reasons:

1. The prospect of having eventual consistency between the various pieces of the pipeline (e.g., the Alpha model having processed some transactions but the later stages having missed them) seems "very bad" to me, though I can't put my finger on exact reasons why. And in any case, this just means my entire application now has 3 (slightly smaller) aggregates vs. 1 all-encompassing aggregate.
2. Unlike "textbook" aggregate cases (e.g., a Products aggregate where each Product is an aggregate), my invariants don't align nicely along sub-dimensions like this. For example, you would think Security might be a consistency boundary, but there are many transactions that cross securities (e.g., a relative value Insight that involves one security to buy and another to sell).

Question

Is there an obvious aggregate design that I'm missing here? If not, is there an analytical framework or tool to help me refine the above aggregate design or to get comfortable with it being the best possible option?

Answer 1

In short

It is probable that in your case the right aggregate design is to use no aggregate at all, because you're not at the a level of abstraction suitable for DDD.

The kind of problems you are dealing with seems to belong to the event streaming family: what is important is that your data points are processed, and once the processing is started, it'll continue to the end.

In this case, eventual consistency allows a bigger scalability: while one processor computes the changes of the last 1000 datapoints to the alpha model, the second processor could propagate the changes caused by the 1000 previous datapoints to the the portfolio model. Transactional consistency would significantly constrain the scalability of the stream processing.

More details

When applying DDD, we break down the domain model in entities and value objects, and group them into consistent aggregates. Let's look at Evans' definitions:

ENTITY: object fundamentally defined not by its attributes, but by a thread of continuity and identity.

VALUE OBJECT: An object that describes some characteristic or attribute but carries no concept of identity.

AGGREGATE: A cluster of associated objects that are treated as a unit for the purpose of data changes. External references are restricted to one member of the AGGREGATE, designated as the root. A set of consistency rules applies within the AGGREGATE’S boundaries.

Let's look at your domain. I understand that adding one or more new Data Points would lead to an update of the Alpha Model and in consequence of the Portfolio Model.

A first question: what would the external reference be for your aggregate? The data point? A group of data points? Or the whole model? If it's the whole model, I understand that in reality you only keep one single such model up-to-date. In this case, what's the benefit of accessing every related objects via the aggregate root entity? Probably none!

A second question: does your Alpha Model have an identity? In other words, do you see the Alpha Model before adding the points the same Alpha Model as after, but with an updated state? Or is it a different model, i.e. the different state with different values make it another model and vice-versa, two Alpha Models with the same state would be completely interchangeable? If this question does not seem relevant, or if you could defend both point of views, then probably you're dealing with some kind of magma. Probably, you should consider smaller subparts as value object or entity, and these would lead you to a different aggregates

Last question: does each and every single Data point require the immediate update of the Alpha Model and the Portfolio Model? Is it important to have a point by point processing. If one of the transaction gets roll back, can you process the following data points or are you stuck? Would it make sense (in theory) to be able to reconstruct the history of the successive model changes? If the answers are no, it's probably because data points could be processed in batches: the point by point transactional consistency would then be a constraint for scalability What really matters is that the Data points are processed in the end and that the changes propagated across the successive Alpha model and Portfolio model. Eventual consistency can guarantee this while at the same time allowing higher scalability and throughput.

One you have recognised the last point, and accepted eventual consistency as part of the solution, you may break down the huge aggregate, using DDD if it adds value to your specific kind of problems. But as said, you'll probably have to work at a more detailed level than alpha model and portfolio model, and you probably have to analyse the computational dependencies between the calculation "behaviors" to get it right.