We want to pull application and business metrics from the source web application to track billing, usage and performance of the application. These metrics are to be stored in a different database (Oracle) for further processing and analytics. We would be building analytics dashboards over these metrics that would be presented to different stakeholders including clients. Below are the points that should be noted
- Metrics collection should have a very low performance overhead (cpu, memory, storage) on the source web application server (java ee based)
- It should not introduce new components in the system infrastructure given that we would be collecting 100-200 metrics from the source application. It may not worth the maintenance effort (deployment/operations/overhead).
- Some of the metrics are event based. For e.g. web service request size, user uploaded file size, user login and logout timestamp etc. Also not all metrics are of data type numeric. For e.g. ip-address, timestamps.
- Metric collection is to be done from around 50-100 deployments (multi-tenant & single-tenant) of the application.
I would like to understand the different architecture approaches that we can take into consideration for collecting these metrics into a different database. Please provide enough details to get an idea of how the implementation would look like.