Reports in microservice architecture

Question

SITUATION

Let's assume we want to generate an excel report with the data from microservice A and present it in microservice B. We can obtain the data with some scheduled task or so.

POSSIBLE SOLUTIONS

Simplest
We add a report functionality to microservice B and generate reports there.
Pros:
Pretty strightforward, on the fly generation (no need to store files)
Cons:
In my opinion it is a violation of bounded context. We mix two functionalities although actually one uses another.

Extracting new microservice dedicated for reports
We create a microservice C which gather data aggregates for reports from A and serve it for B.
Pros:
Separation of responsibilities. In my opinion bounded context is not violated.
Cons:
Either you have to send a generated excel file between microservices (awful idea) or aggregates (not ideal too). The solution for this could be generating a link to the document and send this to B to redirect the user. This seems to be nice idea but on the other hand we need to store the generated document (in opposite to on the fly generation).

What is your experience in simillar situations? Maybe someone has another approach?

UPDATE

I am adding a sketch to better illustrate an actual problem.

Answer 1

OK, This is not a problem of bounded contexts.

You can have more than one service in a bounded context, and its impossible to say what should go in what context when you abstract the functionality to simple names like "A" and "B".

The key thing to realise is that micro services should be micro or even nano

your preference should generally be to add a new service rather than to expand an existing one.

Edit - summary of comments:

The consuming application should make the call directly to the micro-service. If extra info is required, the consuming api should preferably call the requisite micro-services to build that data first, and then send it to the report micro-service.

Cross cutting concerns like authorisation/authentication can be handled by generating a signed token rather than having a single auth service which all the other services depend on.

In this way you avoid building a 'distributed monolith' of services with them all linked and talking to each other.

Answer 2

The solution depends on whether microservice B contributes any data it owns to the report and whether you own microservice A or not. If B does not add any value to the report beyond what's already in A and you have control over A, then it's better to have generation of the report in A. Client may invoke service A directly in this case. Security and navigation have to be taken into account, but the former is responsibility of API gateway and navigation link to the report can be supplied by service B (which will discover it through some internal mechanism).

If B owns some data and/or logic behind the report or you do not own the service A so that you cannot modify it, then fetching any data from A is mere implementation detail and it's perfectly ok to generate report in B with data from A.

Solution with separate microservice C may be too complicated for this task and should be chosen only if B is already big, has a lot of other responsibilities not related to report generation and if there's any value in separate lifecycle for report generation component.

Answer 3

Event Collaboration is a very common solution to that problem.

In your case, microservice A will publish events whenever it changes its state while microservice B is subscribed to some of the events of A in order to generate an aggregate (report). To deliver events from A to B you can use a message bus (like rabbitMq) or event store. In this way, it is going to be very easy to add and extend subscribers without modifying publishers. Also, your system will scale horizontally very well.

A very important thing to keep in mind is that communication should be asynchronous in order to avoid temporal coupling. You can read more about problems connected with synchronous communication in microservices architecture from this article

Answer 4

OK, let first give you my view of a typical microservice architecture, from top to bottom:

• UI consumes microservices through a proxy. The proxy work as a router and aggregator exposing high level API to your microservices
• Proxy layer will aggregate microservices and combine them to fulfill the high level task.
• Microservices. They are fulfilling very small specific tasks, for example Users can be spread in several microservices, typically with an isolated data store.
• Event bus, message queue or something similar, provides infrastructure for message passing, consistency, load balancing and sometimes data replicas.

You can place load balancing, fail over on every layer or the layer you like most.

I think reporting is a very broad domain. You need to cut it up in little pieces first in order to decide where you will put each one. At least, think about the problem as three very broad topics.

1. Real time reporting, that is reports of live data, like list of orders, monitoring data, status. Requires access to real time data.
2. Aggregated or summarized reports in time, that is totals for past periods. Can be read from replicated read-only data.
3. Data mining reports, that is reports that will need to analyze all your past data (archived?). They will require a data warehouse, data lake, you name it. They require to process or preprocess data and store it in a special data store, usually by incremental processing over time.

Now, you see that reporting will need to correlate all the data of different microservices.

• For real time reports, you probably need to place contracts on the needed microservices.
• For Aggregated or summarized reports in time, you can access directly existing read replicas of the needed microservices and deal with the schema change in the reporting layer. That is if the period of read replicas match your reporting needs.
• For data mining, OLAP, you are better getting a data scientist in your reporting team, because you will be dealing with real big data issues.

Depending on the importance of your reporting needs, you will need to spread part in microservices, create some specialized ones, and sync with your data strategy, like how big will your real time data windows would be, how big your historic data window will be, and how often you will be archiving data.

Now, creating the report and generating a PDF or Excel file, belongs to the reporting layer. And I use the word layer on purpose. Reporting will need microservices doing data gathering (probably consuming other microservices, accessing read-only replicas or archived data), report processing, file generation and so on. They need to expose high level interfaces at the proxy layer to be consumed by the UI, such as select a report, fill report parameters, retrieve a file.

Answer

• Create a Reporting layer.
• For data gathering: consume microservices for real time simple reports, as microservices aggregation will increase in complexity and performance will suffer. Create a strategy to access read-only replicas directly. Create a data warehouse, data lake and get a data scientist if dealing with big data. Explore graph databases like Graph Engine www.graphengine.io or Neo4j neo4j.com
• For report processing: build the needed microservices that consume data from your data gathering microservices and use their own isolated data store for the report outcome.
• UI: create your high-level microservices on to the proxy layer.

Answer 5

Does the report generated in A contain more or is it broader then what you will present through B?

Does A exist only for B or are other services using the report also?

If A is only used with B then merge them. Wouldn't make C.

This microservice architecture stuff is obviously leading to potential over fragmentation, you now apply cohesion and separation of concerns at the process level where traditionally you would apply these at the component/class/function level within a process. I think it's healthier to still start there and avoid a proliferation of processes.