6

SITUATION

Let's assume we want to generate an excel report with the data from microservice A and present it in microservice B. We can obtain the data with some scheduled task or so.

POSSIBLE SOLUTIONS

Simplest
We add a report functionality to microservice B and generate reports there.
Pros:
Pretty strightforward, on the fly generation (no need to store files)
Cons:
In my opinion it is a violation of bounded context. We mix two functionalities although actually one uses another.

Extracting new microservice dedicated for reports
We create a microservice C which gather data aggregates for reports from A and serve it for B.
Pros:
Separation of responsibilities. In my opinion bounded context is not violated.
Cons:
Either you have to send a generated excel file between microservices (awful idea) or aggregates (not ideal too). The solution for this could be generating a link to the document and send this to B to redirect the user. This seems to be nice idea but on the other hand we need to store the generated document (in opposite to on the fly generation).

What is your experience in simillar situations? Maybe someone has another approach?

UPDATE

I am adding a sketch to better illustrate an actual problem. enter image description here

  • 1
    The DDD notion of Bounded Context is conceptual. It does not mean that each microservice is necessarily in a different Bounded Context. On the other hand, there is nothing wrong with one Bounded Context getting information from another! So, your question does not have enough information for a good answer. I think you need to add some detail and state your presumptions, which may be faulty and perhaps we can help with (such as what a BC is and what violates a BC). – Erik Eidt Jan 27 '17 at 17:47
  • 1
    Further, if B obtaining and processing data from A is a violation, then how does introducing a third microservice, C, not have that same violation? – Erik Eidt Jan 27 '17 at 17:49
  • @Eric Eidt "there is nothing wrong with one Bounded Context getting information from another!" Definitely true. Never said differently. See the sketch for actual problem. The possible violation is when we aggregate data from A (report aggregates) and keep them along with generation of excel files functionality in B. It is even clearer when we think about other microservice (call it D) which uses reports. – Tomasz Maciejewski Jan 29 '17 at 22:21
3

OK, This is not a problem of bounded contexts.

You can have more than one service in a bounded context, and its impossible to say what should go in what context when you abstract the functionality to simple names like "A" and "B".

The key thing to realise is that micro services should be micro or even nano

your preference should generally be to add a new service rather than to expand an existing one.

Edit - summary of comments:

The consuming application should make the call directly to the micro-service. If extra info is required, the consuming api should preferably call the requisite micro-services to build that data first, and then send it to the report micro-service.

Cross cutting concerns like authorisation/authentication can be handled by generating a signed token rather than having a single auth service which all the other services depend on.

In this way you avoid building a 'distributed monolith' of services with them all linked and talking to each other.

microservices

  • There are actually two sub-problems within this problem. The first one is IMHO bounded context problem; aggregates of data from A leaks from bounded context and when stored in B violate its bounded context (e.g. BC from A can be about Users, films they bought, favorite music, etc. and BC from B about admins who e.g. activate users. Now storing aggregates about number of films the user bought in current month seems to spread BC from A to B. It is not a big problem (and could be treated as a projection of data from A) unless we want to reuse service to generate excel file with that data). – Tomasz Maciejewski Jan 30 '17 at 0:36
  • And about your proposal of adding new service; I would say that in my opinion it is the clearest solution in case of separation of functionalities and not spreading models from BC. Unfortunately this solution has different drawbacks as I mentioned them in my question. – Tomasz Maciejewski Jan 30 '17 at 0:39
  • you only get those drawbacks because you are asking B for the report. you should go direct to C – Ewan Jan 30 '17 at 1:00
  • 2
    hmm, you really should move the auth to its own service and generate a token which can be verified by any service – Ewan Jan 30 '17 at 9:48
  • 1
    I favour this one, because it embraces the concept of separation of concerns best. – Thomas Junk Jan 30 '17 at 10:20
2

The solution depends on whether microservice B contributes any data it owns to the report and whether you own microservice A or not. If B does not add any value to the report beyond what's already in A and you have control over A, then it's better to have generation of the report in A. Client may invoke service A directly in this case. Security and navigation have to be taken into account, but the former is responsibility of API gateway and navigation link to the report can be supplied by service B (which will discover it through some internal mechanism).

If B owns some data and/or logic behind the report or you do not own the service A so that you cannot modify it, then fetching any data from A is mere implementation detail and it's perfectly ok to generate report in B with data from A.

Solution with separate microservice C may be too complicated for this task and should be chosen only if B is already big, has a lot of other responsibilities not related to report generation and if there's any value in separate lifecycle for report generation component.

  • I agree with you Ivan. In my actual case I will do generation of report in B as there is no other usage of this for now. It is definitely the simplest solution. I was curious about other people approaches for similar and possibly more complex situations. – Tomasz Maciejewski Jan 29 '17 at 22:22
1

Event Collaboration is a very common solution to that problem.

In your case, microservice A will publish events whenever it changes its state while microservice B is subscribed to some of the events of A in order to generate an aggregate (report). To deliver events from A to B you can use a message bus (like rabbitMq) or event store. In this way, it is going to be very easy to add and extend subscribers without modifying publishers. Also, your system will scale horizontally very well.

A very important thing to keep in mind is that communication should be asynchronous in order to avoid temporal coupling. You can read more about problems connected with synchronous communication in microservices architecture from this article

  • Although event driven architecture is really nice solution and can help with different (coupling, performance, record history, etc.) issues, it is unrelated to actual problem. It doesn't matter if it will be events, queues, rest, grpc or other technology/design pattern in that scenario. – Tomasz Maciejewski Jan 29 '17 at 22:19
  • Please elaborate why it is unrelated to the problem. I'm not talking about event driven architecture, but only about event collaboration. I would also argue that using events is a pretty good solution to a problem. – IlliakaillI Jan 29 '17 at 22:26
  • It is unrelated because the core of this problem is dual - keeping data from one bounded context AND reusing functionality (in that case pdf generation). It doesn't matter if we will send event to B when data in A changes or poll periodically B with new data from A. Still we have to keep somewhere the aggregates from A (and this could be violation of bounded context). We can think about something like a projection (and separate reporting creates such a projection). – Tomasz Maciejewski Jan 29 '17 at 23:21
  • And yes, we can easily add new subscriber for changes from A in event collaboration. However, we still need to duplicate the excel generation functionality at that scenario. Do you see another way of using event collaboration for that particular problem? Maybe I miss something. – Tomasz Maciejewski Jan 29 '17 at 23:21
  • So, you are saying that both A and B should be able to generate PDFs, is that correct? Regarding aggregates - yes, in pub-subscribe scenario, B should build it's own aggregates to based on events from A in order to make pdf generation possible. If A and B can share the same code that does aggregation and PDF generation - that is great. There will be data duplication for sure, but you will temporally decouple A and B, which is a big win. – IlliakaillI Jan 31 '17 at 23:25
1

OK, let first give you my view of a typical microservice architecture, from top to bottom:

  • UI consumes microservices through a proxy. The proxy work as a router and aggregator exposing high level API to your microservices
  • Proxy layer will aggregate microservices and combine them to fulfill the high level task.
  • Microservices. They are fulfilling very small specific tasks, for example Users can be spread in several microservices, typically with an isolated data store.
  • Event bus, message queue or something similar, provides infrastructure for message passing, consistency, load balancing and sometimes data replicas.

You can place load balancing, fail over on every layer or the layer you like most.

I think reporting is a very broad domain. You need to cut it up in little pieces first in order to decide where you will put each one. At least, think about the problem as three very broad topics.

  1. Real time reporting, that is reports of live data, like list of orders, monitoring data, status. Requires access to real time data.
  2. Aggregated or summarized reports in time, that is totals for past periods. Can be read from replicated read-only data.
  3. Data mining reports, that is reports that will need to analyze all your past data (archived?). They will require a data warehouse, data lake, you name it. They require to process or preprocess data and store it in a special data store, usually by incremental processing over time.

Now, you see that reporting will need to correlate all the data of different microservices.

  • For real time reports, you probably need to place contracts on the needed microservices.
  • For Aggregated or summarized reports in time, you can access directly existing read replicas of the needed microservices and deal with the schema change in the reporting layer. That is if the period of read replicas match your reporting needs.
  • For data mining, OLAP, you are better getting a data scientist in your reporting team, because you will be dealing with real big data issues.

Depending on the importance of your reporting needs, you will need to spread part in microservices, create some specialized ones, and sync with your data strategy, like how big will your real time data windows would be, how big your historic data window will be, and how often you will be archiving data.

Now, creating the report and generating a PDF or Excel file, belongs to the reporting layer. And I use the word layer on purpose. Reporting will need microservices doing data gathering (probably consuming other microservices, accessing read-only replicas or archived data), report processing, file generation and so on. They need to expose high level interfaces at the proxy layer to be consumed by the UI, such as select a report, fill report parameters, retrieve a file.

Answer

  • Create a Reporting layer.
  • For data gathering: consume microservices for real time simple reports, as microservices aggregation will increase in complexity and performance will suffer. Create a strategy to access read-only replicas directly. Create a data warehouse, data lake and get a data scientist if dealing with big data. Explore graph databases like Graph Engine www.graphengine.io or Neo4j neo4j.com
  • For report processing: build the needed microservices that consume data from your data gathering microservices and use their own isolated data store for the report outcome.
  • UI: create your high-level microservices on to the proxy layer.
0

Does the report generated in A contain more or is it broader then what you will present through B?

Does A exist only for B or are other services using the report also?

If A is only used with B then merge them. Wouldn't make C.

This microservice architecture stuff is obviously leading to potential over fragmentation, you now apply cohesion and separation of concerns at the process level where traditionally you would apply these at the component/class/function level within a process. I think it's healthier to still start there and avoid a proliferation of processes.

  • Yes, I totally agree with you that microservice architecture can easily lead to overfragmentation. In my scenario A is core functionality, B is administration part. C would be separation of reporting functionality in case other component of the system would want to get reports as well. For me is quite clear that for now it is enough to keep A and B and not create C. – Tomasz Maciejewski Jan 27 '17 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.