I think you need to start by figuring out how the logging will be consumed. One of the hallmarks of cloud deployments is scaling out capacity, so the concept of ephemeral instances of a microservice or application is important. Even if you scale the app manually, you'll likely have multiple instances of the app running simultaneously.
With that in mind, you have the following constraints on how that logging would be useful:
- Log aggregation: Having one place to view and filter your logs makes maintenance much easier. The alternative is having to log in to multiple instances of your app to figure out the general health of the application is tedious and error prone. Particularly if the instance was automatically replaced by the operations team.
- Ephemeral instances: If an instance of your app can come and go based on demand, your logs will no longer exist. That's where log shipping to your log aggregation server is critical
- Tracing Failures: When you have ephemeral instances, or multiple instances of an app or microservice, you need the ability to reconstruct what happened after the fact. That means you need some additional context.
Splitting Responsibility
Systems like the Elastic Stack, Splunk, DataDog, etc. all have the concept of a log shipper which can provide all the necessary context to your log events so you can trace it to the container or server that the application is running on. They also have the concept of centralized log aggregation.
That means that you can do the following:
- You can add any set of constants to the shipped log entries. For example adding a "role" for a log file so you can use it in search terms later. This seems to fit your intended use.
- Your application only needs to worry about logging its behavior.
As to whether you log to STDOUT or to a file, that really depends on how the application is run. If you are running inside a kubernetes cluster, then K8s will make all STDOUT logs accessible through the container. However if you are running on actual machine instances, those STDOUT logs will be lost. Most logging libraries allow you to customize the output with configuration, so leverage that to make the decision on where the logs end up an operations concern.
Logs are an operational concern
You can easily create a lot of log traffic if you are not careful, and not all of those logs are useful all the time. When you have control over log levels, it's useful to separate what is needed for local debugging vs. what is needed to understand day to day operations. Log levels were intended for this purpose. Caveat: if you are using 3rd party frameworks, you can't control how those tool vendors set up log levels.
When you do have control, it's useful to set your logs like this:
- DEBUG (and lower): should be reserved for software developers, as they are logging low level events to help debug the internal workings while running locally.
- INFO: should be reserved for normal operational activity of your application
- WARNING: should be used for when you detect conditions that may make your app/microservice unstable (nearing memory limits or disk space limits for exaample)
- ERROR: should be used for error reporting that is not expected behavior. Good example: file permission issues preventing the app from saving data. Bad example: parse error that is a normal part of processing data and normally reported back to the user through an API
- SEVERE: unrecoverable errors leading to shutdown
Many logging providers have a variation on a theme here, but you can usually map the concepts above to the library's term for the same concept.
This allows the operations team to lower the volume of log traffic by eliminating the DEBUG and lower messages, and set up alerts on the WARNING and higher messaages.
OpenTracing
It's great that you have a session ID to track user behavior. Including that information in your logs helps track things at a session level. Going a step further than that, many cloud log systems have support for something called Application Performance Monitoring (APM). And one of the most commonly supported standards to support that is OpenTracing.
This allows you to trace a specific request from start to finish, even across asynchronous communications. There's work to be done to define where each context begins and ends, but OpenTracing provides a way to track child contexts with parent contexts. That way the logging aggregator you use can visualize that for you in a dashboard, and you can drill in to the specific log messages that apply.
It's particularly useful for the following reasons:
- You can calculate the overall processing time of every action
- You can detect outliers where the performance is way out of normal
- You can drill in and find out what happened in the outlier
Conclusion
Make sure you know how you are going to consume your logging data, and design for that use case. Know where you can decorate your logs outside of your application so you can divide the responsibilities appropriately. And think about what is necessary to support the application once it's deployed.
Your first step into the process may not be 100% correct, and that's OK. If the app is successful, you'll have the opportunity to add in log messages that are needed, and remove or downgrade the level of log messages that are no longer useful.
