Should we reuse web app logs for user behaviour analytics?

Question

Our web app generates a large amount of logs. These logs include both events regarding background operations in the app (data arrives from the server, ajax failures, inter-component communication, etc.); and also user initiated actions (user clicked a button, user wrote text, etc.).

We built our own logging library with different adapters (print to console, send to server, etc.); and currently send all the logs to our server for persistence. These logs are used to analyze the behaviour and flow of the app, monitor errors, client-side exceptions, etc.

We now have a new requirement of tracking user behaviour in the app, and we consider 2 approaches:

1. Enrich our current in-code logs (which are sent to the server) and log every user action we need to track. Then use ETL jobs to collect and analyze the data using some third party service (Omninute, Kibana, etc.).
2. Integrate a third party service with its own JS library (Omniture, Google Analytics) and adapt our code to use that service (manually sending events from JS, HTML tagging, etc.).

The first approach keeps our code base cleaner and with less duplication (only one logging mechanism).

The second approach involves modifying the app's code to send all the events we want to track to the analytics service, in addition to logging them with our own logging service. But it allows the analytics service to gather additional data which we don't need to implement ourselves (geotracking, browser and OS versions, etc.).

What approach should I take so the code can meet both logging requirements without unnecessary code duplication and complexity?

Answer 1

Do not think about logging as "I want to log this data" think of it as "I have an event that may be interesting to somebody."

The fact that 99% of the time the interested party is an object that takes logging events and then writes to a disk-based log is irrelevant to that mindset.

Logging is essentially a producer-consumer event framework

What you should do is define a characteristic of these "special" events. Perhaps it is a different category, or topic. Then define a consumer, or appender, that logs them in a way that the third-party software can consume. Perhaps this is a separate disk-based log file, or maybe a separate table in a database. Maybe it pipes them through a web service. The implementation is irrelevant, the important part is that they use the same logging framework and the code doing the logging does not need to know about it.

Answer 2

I usually hit three major concerns when building analytics systems.

1. How lossy can the analytics event set be? For example ad blockers tend to block most third party in-browser analytics. Depending on your audience that can be more than 20% of your site visitors. On the other hand more reliable events can impose real performance costs on your systems.
2. How stable does your analytics event schema need to be? The set of logs or events you report will tend to change over time. That can invalidate or at least complicate efforts to compare user behavior. In general the easier it is to report events the harder it will become to reconstruct the context around them for analysis later (e.g. you move a button and users interact with it differently but it reports the same event). You'll have to choose where you're willing to pay that sort of cost.
3. How sensitive is the data you are collecting? It is ever permissible to share with a third party, is some sanitization required first?

The products I have worked on discover they need at least one but often two categories of analytic data and the requirements for them are different enough that I think it's often reasonable to have different systems for reporting them.

Some events are fairly ephemeral. You want to compare split test behaviors or user actions across consecutive versions of an app. Developers usually want these to be automatic or very easy to collect and the queries or reports which use them are often written after data has been collected. It's preferable to err on the side of collecting lots of data than to make data collection expensive and incentivize the team to avoid it. Generally the results can be relative (% of users who completed a funnel) so an evenly distributed loss of events is tolerable. Log ETLs work well for this as do third party JavaScript libraries depending on exactly what activity you want visibility into. Error reporting and diagnostics often fall into this category as well; you're not sure what data you'll need to identify a problem but you'll be sad if you don't have it.

Other events become a core part of the business or service. Revenue numbers are hopefully captured somewhere in your backend already but there's often a need to correlate those precisely with events normally only found in the less well defined bucked above. When you need to answer questions like "which feature generated this revenue" or "which referral campaigns lead to these sales" these can become critical business concerns. If the development team is viewing them as "nice to have", "best effort" measurements where a lossy event stream is acceptable or where schema changes can occur frequently without warning (and without migrating historical data) you hit a painful divide in the organization. This sort of data set probably deserves it's own mechanism for collection where you can introduce a more reliable schema and add validation checks as necessary. This might even be part of an audit log, a case where usage information is expected to be exhaustive.

tl;dr it all depends on your requirements and you need to understand those in detail

Answer 3

I suggest looking at event data aggregators like Segment which can send data to both your backend and several analytics services at the same time. This would provide a solve for both your purposes without duplication of event logging in your code.