We want to pull application and business metrics from the source web application to track billing, usage and performance of the application. These metrics are to be stored in a different database (Oracle) for further processing and analytics. We would be building analytics dashboards over these metrics that would be presented to different stakeholders including clients. Below are the points that should be noted

  1. Metrics collection should have a very low performance overhead (cpu, memory, storage) on the source web application server (java ee based)
  2. It should not introduce new components in the system infrastructure given that we would be collecting 100-200 metrics from the source application. It may not worth the maintenance effort (deployment/operations/overhead).
  3. Some of the metrics are event based. For e.g. web service request size, user uploaded file size, user login and logout timestamp etc. Also not all metrics are of data type numeric. For e.g. ip-address, timestamps.
  4. Metric collection is to be done from around 50-100 deployments (multi-tenant & single-tenant) of the application.

I would like to understand the different architecture approaches that we can take into consideration for collecting these metrics into a different database. Please provide enough details to get an idea of how the implementation would look like.

  • my implementation would look very much like statsd – Ewan Feb 14 '18 at 11:49
  • @Ewan - your suggestion is great but how do we persist these metrics into Oracle after they reach statsd and how do we transfer non-numeric metrics like timestamps & ip-address? – Andy Dufresne Feb 15 '18 at 9:36
  • a: you wouldn't. b: those are probably not metrics, but logs – Ewan Feb 15 '18 at 10:00
  • Point a is a requirement. There are many metrics stored in the database (Oracle) apart from the ones that i need to pull from the app server. Hence i need to persist them in the database. They can be output to a csv file after which we already have data load sql scripts to load them into the database. The question is how do we do it with statsd. Point b - yes these need not be called metrics but the requirement is to collect them. Do you suggest a different mechanism to collect them? Also point c - how do we handle cases where either statsd is down or the database server (Oracle) is down? – Andy Dufresne Feb 15 '18 at 10:12

You have conflicting requirements.

A metric is usually an aggregated calculation, ie "how many times per second was my website hit?"

Databases are not usually suitable for Metrics collection, either you insert a new row per individual event and then run an aggregated sql query, which uses a lot of disc space and cpu, or you continually update a histogram style row, which tends to be slow due to locking.

Something like statsd gets around this with a custom database which does the aggregation for you and UDP fire and forget connections.

The downside is that you can lose individual records, or have them recorded inaccurately if there are network issues, or the server runs out of capacity.

If you have an audit event, which must be recorded, then a database is a good solution, as you can do things like confirm the record is written, roll back transactions when errors happen etc. But all this comes at the cost of speed.

If you want to run aggregated reports on audit records, say for example financial reports. The a good solution is to off load the reporting work to a data warehouse by exporting your transactions to it.

This tends to involve a scheduled "transfer data" task though, so isnt so good for stuff where you want immediate feedback eg. "web server 6 is about to explode!!!"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.