44

I am diving in the domain driven design (DDD) and while I go more deeply in it there are some things that I don't get. As I understand it, a main point is to split the Domain Logic (Business Logic) from the Infrastructure (DB, File System, etc.).

What I am wondering is, what happens when I have very complex queries like a Material Resource Calculation Query? In that kind of query you work with heavy set operations, the kind of thing that SQL was designed for. Doing those calculations inside the Domain Layer and working with a lot of sets in it is like throwing away the SQL technology.

Doing these calculations in the infrastructure can't happen too, because the DDD pattern allows for changes in the infrastructure without changing the Domain Layer and knowing that MongoDB doesn't have the same capabilities of e.g. SQL Server, that can't happen.

Is that a pitfall of the DDD pattern?

  • 34
    While SQL is designed to handle relational set algebra, it is not a fun day when you realize that half of your business logic is buried in a handful of SQL functions that are hard to refactor and even harder to test. So, moving this to the domain layer where it can play with its friends sounds appealing to me. Is this throwing away a good chunk of the SQL technology? Sure, but SQL is a lot easier to manage when you're only using SELECT/JOIN. – Jared Goguen Apr 8 at 13:13
  • 30
    @JaredGoguen but this can be because your not an SQL expert and not because of the technology – Leonardo Mangano Apr 8 at 13:17
  • 2
    @JimmyJames what I tried to say is that if the DDD is well implemented it allows to change layers with the minimum effort, like switching from SQL Server to MongoDB. But, if have complex queries in the SQL, it's possible that I won't be able to switch to MongoDB because their technical differences. I think I said a obvious thing. – Leonardo Mangano Apr 8 at 13:52
  • 7
    ... is like throwing away the SQL technology Just because a particular technology can do something doesn't mean it is the best choice. It's anecdotal evidence, but I've met far too many businesses that used to store business logic in the database and are migrating away from it because of the long term maintainability headaches it causes. Oversimplifying, but databases are meant for storing data and programming languages are meant for transforming data. I wouldn't want to use a DB for business logic anymore than I'd want to try to use my application to store my data directly. – Conor Mancone Apr 8 at 20:23
  • 8
    SQL itself is a great example of DDD. When faced with organizing related data people first specified a language to do it: SQL. Implementation does not really matter much. A DB admin does not need to know C/C++ to query the database. Similarly when faced with the task of scheduling events someone came up with the CRON syntax (m h d m w) a simple domain model that fits 99% of scheduling problems. The core of DDD is not to create classes or tables etc. It is to understand your problem and come up with a system to work in your problem domain – slebetman Apr 9 at 8:33
39

These days, you are likely to see reads (queries) handled differently than writes (commands). In a system with a complicated query, the query itself is unlikely to pass through the domain model (which is primarily responsible for maintaining the consistency of writes).

You are absolutely right that we should render unto SQL that which is SQL. So we'll design a data model optimized around the reads, and a query of that data model will usually take a code path that does not include the domain model (with the possible exception of some input validation -- ensuring that parameters in the query are reasonable).

  • 13
    +1 Good answer, but you should give this concept its proper name, Command-Query Segregation. – Mike Supports Monica Apr 8 at 14:19
  • 6
    @Mike Having completely different models for reading and writing is more like CQRS rather than CQS. – Andy Apr 8 at 15:17
  • 3
    The "read model" is not the domain model (or part of it)? I am not an expert on CQRS, but I always thought the command model is quite different from the classic domain model, but not the read model. So maybe you can give an example for this? – Doc Brown Apr 8 at 17:14
  • Took me too long to realize that High Performance Mark was calling attention to a typo. – VoiceOfUnreason Apr 10 at 11:50
  • @DocBrown - here is my attempt to clarify for you --> cascadefaliure.vocumsineratio.com/2019/04/… – VoiceOfUnreason Apr 11 at 4:03
21

As I understand it, a main point is to split the Domain Logic (Business Logic) from the Infrastructure (DB, File System, etc.).

This is the foundation of the misunderstanding: the purpose of DDD isn't to separate things along a hard line like "this is in the SQL server, so must not be BL", the purpose of DDD is to separate domains and create barriers between them that allow the internals of a domain to be completely separate from the internals of another domain, and to define shared externals between them.

Don't think of "being in SQL" as the BL/DL barrier—that's not what it is. Instead, think of "this is the end of the internal domain" as the barrier.

Each domain should have external-facing API's that allow it to work with all the other domains: in the case of the data storage layer, it should have read/write (CRUD) actions for the data-objects it stores. This means SQL itself isn't really the barrier, the VIEW and PROCEDURE components are. You should never read directly from the table: that is the implementation detail DDD tells us that, as an external consumer, we should not worry about.

Consider your example:

What I am wondering is, what happens when I have very complex queries like a Material Resource Calculation Query? In that kind of query you work with heavy set operations, the kind of thing that SQL was designed for.

This is exactly what should be in SQL then, and it's not a violation of DDD. It's what we made DDD for. With that calculation in SQL, that becomes part of the BL/DL. What you would do is use a separate view / stored procedure / what-have-you, and keep the business logic separated from the data-layer, as that is your external API. In fact, your data-layer should be another DDD Domain Layer, where your data-layer has it's own abstractions to work with the other domain layers.

Doing these calculations in the infrastructure can't happen too, because the DDD pattern allows for changes in the infrastructure without changing the Domain Layer and knowing that MongoDB doesn't have the same capabilities of e.g. SQL Server, that can't happen.

That's another misunderstanding: it says implementation details internally can change without changing other domain layers. It doesn't say you can just replace a whole infrastructure piece.

Again, keep in mind, DDD is about hiding internals with well-defined external API's. Where those API's sit is a totally different question, and DDD doesn't define that. It simply defines that these API's exist, and should never change.

DDD isn't setup to allow you to ad-hoc replace MSSQL with MongoDB—those are two totally different infrastructure components.

Instead, let's use an analogy for what DDD defines: gas vs. electric cars. Both of the vehicles have two completely different methods for creating propulsion, but they have the same API's: an on/off, a throttle/brake, and wheels to propel the vehicle. DDD says that we should be able to replace the engine (gas or electric) in our car. It doesn't say we can replace the car with a motorcycle, and that's effectively what MSSQL → MongoDB is.

  • 1
    Thanks for the explanation. For me is a very hard topic, everyone have a different point of view. The only thing I don't agree is the comparison between MSSQL(car) and MongoDB (motorcycle), for me the right comparison is that these are two different engines for the same car, but it's just a opinion. – Leonardo Mangano Apr 8 at 15:03
  • 8
    @LeonardoMangano Ah, but they're not. MSSQL is a relational database, MongoDB is a document database. Yes, "database" describes both, but that's about as far as it goes. The read/write techniques are completely different. Instead of MongoDB, you could use Postgre or MySQL as an alternative, and that would be a valid comparison. – 410_Gone Apr 8 at 15:05
  • 3
    "You should never read directly from the table..." Madness. – jpmc26 Apr 9 at 1:20
  • "You should never read directly from the table..." This is a rule that I've come to implement on my own after a decade of writing software that interfaces with databases and suffering through the early pain of trying to follow tutorials structured around popular design patterns. – Lucifer Sam May 7 at 12:28
  • @LuciferSam Aye. It makes it far easier to manage the separation between implementation details, and domain boundaries. One "object" in the domain might be represented by 5 tables, so we use a View to encapsulate that object. – 410_Gone May 7 at 13:25
18

If you've ever been on a project where the organization paying to host the application decides that the database layer licenses are too expensive, you'll appreciate the ease of which you can migrate your database/data storage. All things considered, while this does happen, it doesn't happen often.

You can get the best of both worlds so to speak. If you consider performing the complex functions in the database an optimization, then you can use an interface to inject an alternate implementation of the calculation. The problem is that you have to maintain logic in multiple locations.

Deviating from an architectural pattern

When you find yourself at odds with implementing a pattern purely, or deviating in some area, then you have a decision to make. A pattern is simply a templated way to do things to help organize your project. At this point take time to evaluate:

  • Is this the right pattern? (many times it is, but sometimes it's just a bad fit)
  • Should I deviate in this one way?
  • Just how far have I deviated so far?

You'll find that some architectural patterns are a good fit for 80-90% of your application, but not so much for the remaining bits. The occasional deviation from the prescribed pattern is useful for performance or logistical reasons.

However, if you find that your cumulative deviations amount to a good deal more than 20% of your application architecture, it's probably just a bad fit.

If you choose to keep going with the architecture, then do yourself a favor and document where and why you deviated from the prescribed way of doing things. When you get a new enthusiastic member on your team, you can point them to that documentation which includes the performance measurements, and justifications. That will reduce the likelihood of repeat requests to fix the "problem". That documentation will also help disincentivize rampant deviations.

  • I would avoid the use of phrases like "is this the right pattern" in answers. It's hard enough to get people to be specific when they write their questions, and by your own admission "sometimes it's a bad fit," which suggests that no, it's not the right pattern. – Robert Harvey Apr 8 at 14:49
  • @RobertHarvey, I've been in projects where the pattern used was just not right for the application, which caused it to fail certain quality metrics. It's certainly not the norm, but when that happens you have the hard decision to change architectures or keep shoehorning code into the app. The sooner you can determine the bad fit, the easier it is to fix. That's why I always include that thought while evaluating edge cases. Along with the last bullet, sometimes you don't realize how bad a fit it is until you see the accumulation of deviations. – Berin Loritsch Apr 9 at 12:41
7

The set manipulation logic that SQL is good at can be integrated with DDD no problem.

Say for example I need to know some aggregate value, total count of product by type. Easy to run in sql, but slow if I load every product into memory and add them all up.

I simply introduce a new Domain object,

ProductInventory
{
    ProductType
    TotalCount
    DateTimeTaken
}

and a method on my repository

ProductRepository
{
    List<ProductInventory> TakeInventory(DateTime asOfDate) {...}
}

Sure, maybe I am now relying on my DB having certain abilities. But I still technically have the separation and as long as the logic is simple, I can argue that it is not 'business logic'

  • Well, so far I recall. Repositories are supposed to get Query as parameters too. repository.find(query);. I have read the same but with Specs. That opens a door to leave Query` as an abstraction and QueryImpl or the specific-query implementation to the infrastructure layer. – Laiv Apr 8 at 14:19
  • 5
    oh god, I know some people do that but I think it's awful. You can view this kind of thing as a step down that road. But I think it can be taken with caution. – Ewan Apr 8 at 14:23
  • I know some people do that some people are Pivotal and its framework. SpringFramework has a lot of this:-). Anyways, As @VoiceOfUnreason has suggested, the key around DDD is keeping the consistency of the writings. I'm unsure about forcing the design with domain models whom only purpose is querying or parametrizing queries. That could be approached out of the domain with data structures (pocos, pojos, dtos, row mappers, whatever). – Laiv Apr 8 at 14:28
  • obviously we need some sort of inquisition to help those people back to sanity. But I'm sticking to my guns. The partial exposure of the datalayer is acceptable when it objectively makes for a better application, where what is or isn't a "Domain Object" is subjective – Ewan Apr 8 at 14:35
  • 1
    @LeonardoMangano depends on your application and implementation. The main thing to realise is that you can reinterpret your Domain to make it practicable. – Ewan Apr 8 at 14:57
3

One of the possible ways to solve this dilemma is to think of SQL as of an assembly language: you rarely, if at all, code directly in it, but where performance matters, you need to be able to understand the code produced by your C/C++/Golang/Rust compiler and maybe even write a tiny snippet in assembly, if you cannot change the code in you high level language to produce desired machine code.

Similarly, in realm of databases and SQL, various SQL libraries (some of which are ORM), e.g. SQLAlchemy and Django ORM for Python, LINQ for .NET, provide higher level abstractions yet use generated SQL code where possible to achieve performance. They also provide some portability as to the used DB, possibly having different performance, e.g. on Postgres and MySQL, due to some operations using some more optimal DB-specific SQL.

And just as with high level languages, it is critical to understand how SQL works, even if it is just to rearrange the queries done with above mentioned SQL libraries, to be able to achieve desired efficiency.

P.S. I would rather make this a comment but I do not have sufficient reputation for that.

2

As usual, this is one of those things that depends on a number of factors. It's true that there's a lot that you can do with SQL. There are also challenges with using it and some practical limitations of relational databases.

As Jared Goguen notes in the comments, SQL can be very difficult to test and verify. The main factors that lead to this are that it can't (in general) be decomposed into components. In practice, a complex query must be considered in toto. Another complicating factor is that be behavior and correctness of SQL is highly dependent on the structure and content of your data. This means that testing all the possible scenarios (or even determining what they are) is often infeasible or impossible. Refactoring of SQL and modification of database structure is likewise problematic.

The other big factor that has lead to moving away from SQL is relational databases tend to only scale vertically. For example, when you build complex calculations in SQL to run in SQL Server, they are going to execute on the database. That means all of that work is using resources on the database. The more that you do in SQL, the more resources your database will need both in terms of memory and CPU. It's often less efficient to do these things on other systems but there's no practical limit to the number of additional machines you can add to such a solution. This approach is less expensive and more fault-tolerant than building a monster database server.

These issues may or may not apply to the problem at hand. If you are able to solve your problem with available database resources, maybe SQL is fine for your problem-space. You need to consider growth, however. It might be fine today but a few years down the road, the cost of adding additional resources may become a problem.

  • Isn't the alternative to a monster database, simply a monstrous number and diversity of auxiliary systems? What resilience do the auxiliary systems have, if they all hang off the core system? And if the justification is simply the technological limitation of the core system, then this will often be a premature optimisation for most business systems. SQL can in general be written in a decoupled fashion, if deemed necessary. – Steve May 29 at 11:39
  • @Steve I think where you've gone wrong here is assuming there must be a single core system that others 'hang off' of. – JimmyJames May 29 at 13:39
  • @Steve To give an example, you can replace a systems entire database with a single no-SQL database (i'm not saying this is always the right choice, just that it can be done.) That database can then be stored across many systems, even geographic regions. Such a DB is not auxiliary, it's a wholesale replacement of the SQL DB. – JimmyJames May 29 at 13:58
  • @JimmyJames, agreed, but when there isn't a core system, that can create its own problems analysing dependencies and maintaining data consistency. That's the reason for monoliths in the first place - they create a certain kind of simplicity, and therefore certain kinds of analysis and maintenance efficiencies. Non-monolithic solutions merely exchange some problems or costs for others. – Steve May 29 at 13:59
  • @jmoreno Throwing resources at something in order to limp along with it is not what I would call good engineering: "in order to handle the site’s massive data volume, and is running 9,000 instances of memcached in order to keep up with the number of transactions the database must serve." Do you consider the cost of your designs or do you assume someone will shell out money to make your personal preferences workable? – JimmyJames May 29 at 14:04
2

Is that a pitfall of the DDD pattern?

Let me first clear a few misconceptions.

DDD is not a pattern. And it doesn't really prescribe patterns.

The preface to Eric Evan's DDD book states:

Leading software designers have recognized domain modeling and design as critical topics for at least 20 years, yet surprisingly little has been written about what needs to be done or how to do it. Although it has never been formulated clearly, a philosophy has emerged as an undercurrent in the object community, a philosophy I call domain-driven design.

[...]

A feature common to the successes was a rich domain model that evolved through iterations of design and became part of the fabric of the project.

This book provides a framework for making design decisions and a technical vocabulary for discussing domain design. It is a synthesis of widely accepted best practices along with my own insights and experiences.

So, it's a way to approach software development and domain modeling, plus some technical vocabulary that supports those activities (a vocabulary that includes various concepts and patterns). It's also not something completely new.

Another thing to keep in mind is that a domain model is not the OO implementation of it that can be found in your system - that's just one way to express it, or to express some part of it. A domain model is the way you think about the problem you are trying to solve with the software. It's how you understand and perceive things, how you talk about them. It's conceptual. But not in some vague sense. It's deep and refined, and is a result of hard work and knowledge gathering. It is further refined and likely evolved over time, and it involves implementation considerations (some of which may constrain the model). It should be shared by all team members (and involved domain experts), and it should drive how you implement the system, so that the system closely reflects it.

Nothing about that is inherently pro- or anti-SQL, although OO developers are perhaps generally better at expressing the model in OO languages, and the expression of many domain concepts is better supported by OOP. But sometimes parts of the model must be expressed in a different paradigm.

What I am wondering is, what happens when I have very complex queries [...]?

Well, generally speaking there are two scenarios here.

In the first case, some aspect of a domain really requires a complex query, and perhaps that aspect is best expressed in the SQL/relational paradigm - so use the appropriate tool for the job. Reflect those aspects in your domain thinking and the language used in communicating concepts. If the domain is complex, perhaps this is a part of a subdomain with it's own bounded context.

The other scenario is that the perceived need to express something in SQL is a result of constrained thinking. If a person or a team has always been database oriented in their thinking, it may be difficult for them, just due to inertia, to see a different way of approaching things. This becomes a problem when the old way fails to meet the new needs, and requires some thinking out of the box. DDD, as an approach to design, is in part about ways to find your way out of that box by gathering and distilling the knowledge about the domain. But everybody seems to ignore that part of the book, and focuses on some of the technical vocabulary and patterns listed.

0

Sequel became popular when memory were expensive, because relational data model provided possibility to normalise your data and effectively store it in the file system.

Now memory is relatively cheap, so we can skip normalisation and store in in the format we use it or even duplicate a lot of same data for sake of speed.

Consider database as simple IO device, which responsibility to store data in the file system - yes I know it is difficult to imagine it, because we wrote plenty of applications with important business logic written into SQL queries - but just try to imagine that SQL Server is just another printer.

Would you embedded PDF generator into printer driver or added a trigger which will print log page for every sales order printed out of our printer?

I assume the answer will be no, because we don't want that our application are coupled to the specific device type (not even talking about efficiency of such idea)

In 70's- 90's SQL database were efficient, now? - Not sure, in some scenarios asynchronous data query will returns required data faster than multiple joins in SQL query.

SQL wasn't designed for complicated queries, it were designed for storing data in efficient way and then provide interface/language to query stored data.

I would say building your application around relational data model with complicated queries is abuse of database engine. Of course database engine providers are happy when you tightly coupling your business to their product - they will be more than happy to provide more features which make this bound stronger.

  • 1
    But I keep thinking that SQL is way better for set calculations than any other language. From my point of view. your example is upside down, using C# for very complex set operations with million of rows and joins involved is using the wrong tool, but I could be wrong. – Leonardo Mangano Apr 9 at 21:13
  • @LeonardoMangano, some examples: with c# I can chunk millions of rows and calculate it in parallel, I can retrieve data asynchronously and execute calculations "in time" when data is returned, with c# I can do calculations with low memory usage by enumerating row by row. Having complex logic in the code will provide you plenty of options how to do calculations. – Fabio Apr 9 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.