3

I'd like to know what are common architectural pattern for the following problem.

Web application A has information on sales, users, responsiveness score, etc. Some of this information are computationally intensive and or have a complex business logic (e.g. responsiveness score).

I'm building a separate application (B) for internal admin tasks that modifies data in web application A and report on data from web application A.

For writing I'm planning to use a restful api. E.g. create a new entity, update entity, etc.

In application B I'd like to show some graphs and other aggregate data for the previous 12 months. I'm planning to store the aggregate data for each month in redis.

Some data should update more often, e.g every 10 minutes.

I can think of 3 ways of doing this.

  1. A scheduled task in app B that connects to an api of app A that provides some aggregated data. Then app B stores it in Redis and use that to visualise pages. Cons: it makes complex calculation within a web request, requires lot's of work e.g. api server and client, storing, etc., pros: business logic still lives in app A.

  2. A scheduled task in app A that aggregates data in an non-web process and stores it directly in Redis to be accessed by app B.

  3. A scheduled task in app A that aggregates data in a non-web process and uses an api in app B to save it.

I'd like to know if there is a well known architectural solution to this type of problems and if not what are other pros/cons for the solution I've suggested?

  • Sharing your research helps everyone. Tell us what you've tried and why it didn’t meet your needs. This demonstrates that you’ve taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask – gnat May 27 '14 at 16:43
  • There is huge field that tries to answer your questions called OLAP. – Euphoric May 27 '14 at 17:47
  • Anything else I can do to further improve this question? – soulnafein May 28 '14 at 15:19
  • Not sure why it's put on hold. The implementation choice maybe broad, but there's obvious solution to this. It's common to copy data to different database (you can call it data warehouse) using ETL tools. Then you build your dashboard on top of that and/or OLAP (which will do aggregations for you). – imel96 May 29 '14 at 0:43
  • @imel96 question was heavily edited after it was put on hold; here is original revision – gnat May 30 '14 at 11:02
0

I would do it the following way:

Using and ETL tool and scheduled task for example, I'd get the data from app A, and store it in it's most raw form.

From that raw data, I'd use app B to do all of the heavy calculations and displaying.

This keeps the data available and ready to manipulate the way you want it, without having to use app A all of the time.

Keep the applications separate, and just store and use the data as needed, for simplicity.

1

As I mentioned in comment, I'd use ETL tool to extract data from app A database and maybe from the logs too. At the same time, I'd transform the raw data into data cube, I'm guessing for dashboard you will need to mix them with time dimension.

The output from ETL process should be a more structured data that will help you build dashboard or feed them to OLAP tools for analysis.

In operation, the tool would be scheduled to run different tasks depending on your needs, some could be hourly, daily or every 10 mins. If you're concern about the impact on production database, you can setup a database replica to read from.

ps. Even if you still want to create an API, ETL tool can do that for you too, that is read data and present it as API with output in xml/json.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.