In a CQRS/ES application, can projections be dependant?

Question

With event sourcing, you can project an event to create query-optimised read models. This I understand. What I'm unsure about is whether these read models can depend on each other?

I'm considering pre-generating HTML pages and PDF reports in a certain part of an application. The HTML pages are not interactive, but rather information, just like the PDF reports. However, in order to generate the reports, the data needs to be obtained from another projection. Namely, an SQL projection. So the architecture of the application would look at follows:

 +-----------+
 |Event Store|                              +------------+
 +--^-----+--+                              |SQL Database+-------+
    |     |                                 +------^-----+       |
    |     |                                        |             |
    |     |                                        |             |
+---+-----v---+        +------------+         +----+----+        |
|Command Model+-------->Event Stream+----+---->Projector|        |
+-------------+        +------------+    |    +---------+        |
                                         |                       |
                                         |                       |
                                         |    +-------------+    |
                                         +---->HTML Renderer<----+
                                         |    +-------------+    |     +---------+
                                         |           +---------------->+HTML Page|
                                         |                       |     +---------+
                                         |    +------------+     |
                                         +---->PDF Renderer<-----+
                                              +------------+           +------------+
                                                     +---------------->+PDF Document|
                                                                       +------------+

The problem with the above is that the HTML Renderer and PDF Renderer are dependant on another projection, which means that the order in which the projections are built becomes important. Is this a significant problem?

The alternative to the above is to:

1. Treat the HTML/PDF rendering as a query, performed per request.
2. Make the HTML and PDF projectors use their own internal data structures (i.e., an SQL table similar to the SQL projector), removing the ordering issue.
3. Treat the HTML/PDF rendering as a separate layer "on top of" the query layer. E.g., a query says "give me the objects matching criteria C and format it using formater F". The query gets the data from the database, and the appropriate renderer is used to produce output.

Answer 1

Is this a significant problem?

Maybe? How important are differences in timing between the different projections? Assuming that these projections are listening to the events independently of each other, there can be a bit of drift when the projector gets ahead of, or falls behind, the html renderer. What happens to the html renderer if the database is unavailable? Or extremely far behind because we have thrown out the old corrupted copy and are now replaying the entire event stream in the projector?

Treat the HTML/PDF rendering as a query, performed per request.

That doesn't really help with the synchronization issues. But stand alone it's not a bad idea -- caching responses to http requests is fairly well understood.

Make the HTML and PDF projectors use their own internal data structures (i.e., an SQL table similar to the SQL projector), removing the ordering issue.

Not a bad thing, especially if the schema in the database doesn't match the html use case very well.

Another possibility would be to treat the operations like a batch process; the projections, rather than reading from the fire hose as quickly as they can, instead talk to a coordinator, who grants permission to read only to the point that things won't get screwed up.

Answer 2

If I understand your diagram correctly:

• Events from your Event Stream are pushed to the Projector (Doesn't matter if it's a read-style push, like in Kafka).
• The projector then imperatively mutates the SQL database.
• But some of the events also go to the rendering projectors, which causes them to update some kind of (or model used for fast generation of) report.
• But some of these updates have insufficient information so they need to go query the SQL projection. (This is the part I am unsure I understand correctly. If I'm right, the arrow here has a quite different meaning to the rest of the arrows, since instead of the SQL pushing anything to these services when IT wants to, the services come query sql once prodded by an event.)
• But that creates a race condition between the event they are reacting to and the state of the SQL database they will see.

If that is correct, but having the HTML projector contain its own state that it would otherwise get from the SQL projection would lead to too much code/infrastructure duplication, you might want to do the following instead:

Have your Projector, whenever it would update the relevant SQL values, emit an event (a message, not a persisted domain event) to the renderers. There are at least two approaches to this:

1. Have the event reference the event that caused that change. That way, the consuming services can correlate(pair up) the original event and the resulting SQL-processed message/event, and have all the information needed to make a consistent report. They can just drop all halves that are older than the newest complete pair, whether they were already holding on to them or they are incoming now.
2. Have the Projector send all the information required to update the render projections in one message, consisting of both of part of the data it painstakingly computed based on the SQL "cache" and part of the original event data that wasn't preserved in the SQL.

Finally, in addition to this being part of Projector, it could be a separate service tailing the DB, especially if ALL the information to formulate the message for render service is available in the SQL. In option 2 and this case, the render service won't necessarily be subscribed to the Event Stream at all.