Architecture for dashboard showing aggregated stats

Question

I'd like to know what are common architectural pattern for the following problem.

Web application A has information on sales, users, responsiveness score, etc. Some of this information are computationally intensive and or have a complex business logic (e.g. responsiveness score).

I'm building a separate application (B) for internal admin tasks that modifies data in web application A and report on data from web application A.

For writing I'm planning to use a restful api. E.g. create a new entity, update entity, etc.

In application B I'd like to show some graphs and other aggregate data for the previous 12 months. I'm planning to store the aggregate data for each month in redis.

Some data should update more often, e.g every 10 minutes.

I can think of 3 ways of doing this.

1. A scheduled task in app B that connects to an api of app A that provides some aggregated data. Then app B stores it in Redis and use that to visualise pages. Cons: it makes complex calculation within a web request, requires lot's of work e.g. api server and client, storing, etc., pros: business logic still lives in app A.

2. A scheduled task in app A that aggregates data in an non-web process and stores it directly in Redis to be accessed by app B.

3. A scheduled task in app A that aggregates data in a non-web process and uses an api in app B to save it.

I'd like to know if there is a well known architectural solution to this type of problems and if not what are other pros/cons for the solution I've suggested?

Answer 1

I would do it the following way:

Using and ETL tool and scheduled task for example, I'd get the data from app A, and store it in it's most raw form.

From that raw data, I'd use app B to do all of the heavy calculations and displaying.

This keeps the data available and ready to manipulate the way you want it, without having to use app A all of the time.

Keep the applications separate, and just store and use the data as needed, for simplicity.

Answer 2

As I mentioned in comment, I'd use ETL tool to extract data from app A database and maybe from the logs too. At the same time, I'd transform the raw data into data cube, I'm guessing for dashboard you will need to mix them with time dimension.

The output from ETL process should be a more structured data that will help you build dashboard or feed them to OLAP tools for analysis.

In operation, the tool would be scheduled to run different tasks depending on your needs, some could be hourly, daily or every 10 mins. If you're concern about the impact on production database, you can setup a database replica to read from.

ps. Even if you still want to create an API, ETL tool can do that for you too, that is read data and present it as API with output in xml/json.