performance suggestions on Aggregate root containing thousands of child entities

Question

I understand that DDD is a design concept and implementation through ORMs are tricky but help me understand how would you solve this problem?

So here is my confusion on changing on aggregates only though aggregate roots. Given my case(rehashed through JIRA case). I feel this aggregate and DDD feels good about how we modify this and simplifies logic.

But my question is performance. Just to add a comment to a sub-task we are loading en entire aggregate root into memory. How can this be efficient?

So we have for example An aggregate root Epics which contains Story and that contains sub-tasks.

Lets assume For 1 Epic there could be about 1000 Stories and for each of the Stories there could be 10000 sub-tasks

Now if i want to add a comment to a sub-task or add a comment to story I need to do this

Put Request

api/lates/epics/{epicid}/stories/{storyid}/subtask/{subtaskid}/comment

api/lates/epics/{epicid}/stories/{storyid}/comment

Implement EpicModificationController

@Autowired
EpicModificationService epicModificationService;
........
EpicID epicId = new EpicID(long)
StoryId storyId = new StoryID(long)
SubtaskId subtaskId = new SubtaskID(long)
Comment comment = new Comment("comment text")
.......
epicModificationService.addSubtaskComment(epicID, storyId, subtaskId, comment)

EpicModificationService

Epic epic = epicRepository.findOne(epicId);
epic.addSubtaskComment(storyId, subtaskId, comment);
epicRepository.saveandflush(epic);

Domain Entity

@Entity
class Epic {

  addSubtaskComment(StoryId, SubtaskId, Comment) {
     Story story = this.getStory(StoryId);
     Subtask subtask = story.getSubtask(subtaskId)
     subtask.addComment(Comment);

  }

}

I can directly go fetch my subtask through DAO and update it?

Sounds to be more performant?

I am not there at the level of CQRS + DDD yet as I am yet to understand nuances in DDD itself.

There are reasons why my domain model in correct in way because some actions on Stories cannot be done when Epics are in some state. E.g) you cannot edit a story (add subtask, change 'status' etc , comments are ok) when an Epic in closed. [again Epic/story e.g is just a representation of my real case]

Answer 1

Leaky Abstractions and Abstraction Bypasses

When the abstraction breaks, and bypassing it feels prefferable, its time to fix the abstraction.

• Find a cheap way to speed it up, and put off actually fixing it.
• Adjust the abstraction to provide an optimised redline path for that specific operation.
• Fix the abstraction generically
• Dismantle it entirely and expose everything below that abstraction.

Cheap Fix

Find a cheap solution that doesn't fix the actual issue but hides it.

• Hardware
• Optimising compiler
• Query optimisation
• data rearrangement
• caching

Not the best option from a technical point of view because the problem doesn't go away, it just staves off the problem, and more often than not introduces new problems.

Either way its an interim solution at best, when the load increases you will be back were you started without an easy solution.

Special Case

There are times when a really important business operation is performed many times within tight time constraints. It often makes sense to perform these operations at a lower level.

This will quite likely provide a significant speed up to execution speed. You can optimise it heavily, and remove many layers of dependecy and abstraction.

The problem is that this breaks the way the interface works. All operations on stories occur on story objects, except for this case here which requires you call this global function and pass in some arcane strings. Oh and this case where you can get the story directly, etc...

There is no reason why you cannot offer global functions, or direct accessors, etc.. - If offered as an interface wide idiom.

Offered as special cases they complicate an API, or an abstraction. The complexity leads to subtle bugs. Afterall in ten months time who is going to remember this special case? And when the domain object behaviour changes who is going to update the optimised procedure (assuming they remember about it)?

Lazy Loading

Lets take a look at that abstraction. Why does the Epic have to load all of its Stories?

I see no reason why an aggregate root has to load all or even any of its children, or other associated entities.

Obviously if you have a mechanism for loading all of them in a tolerable amount of time, eager loading is much simpler to implement. This does not sound to be your case though.

Lazy loading is particularly useful if you do not have to even access the database to create the domain entity. Consider the tree of Epic, Story, Task, if you can describe the path (identifiers or selectors) you can access any object on that tree implicitly. Exactly like a FileSystem Path lets you address a file without loading it first, or a CSS selector can target applicable tags in an html document.

Of course the modifying/access operations will have to load/manipulate data but that decision now has the benefit of knowing what you wish to do.

As an added benefit this will change the way you handle database integration, and if done right will bring this same advantage to all of the other domain entities.

Throw it away

Caution: this is a dangerous activity.

Simply throw that abstraction away, its broken, you've tried to fix it but simply cannot. Sometimes it is best to pull it back look at what that abstraction was hiding and try to build a better abstraction over it.

• On the upside you will get the chance to write a better abstraction that satisfies your needs.

• On the downside be prepared for this to seriously slow the project, and possibly even to cause it to fail entirely.

If you want to pursue this option, try to instead build the new abstraction alongside the old abstraction. Progressively swap over old code to use the new interface, duplicating functions as necessary. Once fully migrated delete all the old code.

• On the upside the project can still release, and you can get those improvements in quickly.
• On the downside the project has two interfaces, and duplicate code making it brittle and slower to work with during the transition.

Answer 2

But my question is performance. Just to add a comment to a sub-task we are loading en entire aggregate root into memory. How can this be efficient?

It isn't. Don't do that. This is not a question of DDD. The design of your abstractions and logical model do not dictate the implementation. If you think that DDD requires that you load everything into memory, you have fundamentally misunderstood it. This idea that you must 'load everything upfront' is one of the worst myths in Java-land. When people tell you Java is slow, it's probably because they used an app written in Java designed this way. The mistake is rooted in the idea that once everything is in memory, it will be fast to navigate. The fundamental flaw is that you will load far more than the user will ever need and they will be waiting for a very long time while the application does this to get to the one item they want. Presuming it's successful: you need far more available heap to load everything and as the data grows, your app will likely start crashing with OOME and or drive GC constantly.

The easiest way to deal with this in the existing model is to load only the listing of each level's children as they are needed. That is, when you load an epic, you only load the name, description and any other detail needed by the client to determine which child they want to view. Then when the use selects the Story, you load the listing of the Tasks. Once you have that basic model in place you can start adding caching and predictive loading if you want even more speed. But typically, this will be good enough. It will scream compared to loading everything.