0

I am looking for an advice (or an example) on how to organize logging streams in my cloud app (C#).

In my app there are logging events related to the application infrastructure level and I want to write them to a file (or STDOUT). I use Serilog and I will enrich those events with custom metadata like environment parameters, kubernetes labels etc. These logs are for backend engineers (me).

Besides that I want to have a detailed log of user actions which is very verbose and I want to write it to Loki (or any other log management system). These logs are for user support and analysts. We do not want to pay for a lot of meta information that we do not use. User events a correlated by session ID and there is no need to have a lot of system attributes like kubernetes labels.

So the challenge is to provide specific set of custom properties and specific logging sinks (aka logger providers) for different sources of logging events.

Besides that, common logging libraries like Serilog or Microsoft.Extensions.Logging have a notion of some global logging context (or a scope) which is shared by all loggers within the process. So adding custom properties to a global logging context is not an option.

What do you think?

2 Answers 2

0

I think you need to start by figuring out how the logging will be consumed. One of the hallmarks of cloud deployments is scaling out capacity, so the concept of ephemeral instances of a microservice or application is important. Even if you scale the app manually, you'll likely have multiple instances of the app running simultaneously.

With that in mind, you have the following constraints on how that logging would be useful:

  • Log aggregation: Having one place to view and filter your logs makes maintenance much easier. The alternative is having to log in to multiple instances of your app to figure out the general health of the application is tedious and error prone. Particularly if the instance was automatically replaced by the operations team.
  • Ephemeral instances: If an instance of your app can come and go based on demand, your logs will no longer exist. That's where log shipping to your log aggregation server is critical
  • Tracing Failures: When you have ephemeral instances, or multiple instances of an app or microservice, you need the ability to reconstruct what happened after the fact. That means you need some additional context.

Splitting Responsibility

Systems like the Elastic Stack, Splunk, DataDog, etc. all have the concept of a log shipper which can provide all the necessary context to your log events so you can trace it to the container or server that the application is running on. They also have the concept of centralized log aggregation.

That means that you can do the following:

  • You can add any set of constants to the shipped log entries. For example adding a "role" for a log file so you can use it in search terms later. This seems to fit your intended use.
  • Your application only needs to worry about logging its behavior.

As to whether you log to STDOUT or to a file, that really depends on how the application is run. If you are running inside a kubernetes cluster, then K8s will make all STDOUT logs accessible through the container. However if you are running on actual machine instances, those STDOUT logs will be lost. Most logging libraries allow you to customize the output with configuration, so leverage that to make the decision on where the logs end up an operations concern.

Logs are an operational concern

You can easily create a lot of log traffic if you are not careful, and not all of those logs are useful all the time. When you have control over log levels, it's useful to separate what is needed for local debugging vs. what is needed to understand day to day operations. Log levels were intended for this purpose. Caveat: if you are using 3rd party frameworks, you can't control how those tool vendors set up log levels.

When you do have control, it's useful to set your logs like this:

  • DEBUG (and lower): should be reserved for software developers, as they are logging low level events to help debug the internal workings while running locally.
  • INFO: should be reserved for normal operational activity of your application
  • WARNING: should be used for when you detect conditions that may make your app/microservice unstable (nearing memory limits or disk space limits for exaample)
  • ERROR: should be used for error reporting that is not expected behavior. Good example: file permission issues preventing the app from saving data. Bad example: parse error that is a normal part of processing data and normally reported back to the user through an API
  • SEVERE: unrecoverable errors leading to shutdown

Many logging providers have a variation on a theme here, but you can usually map the concepts above to the library's term for the same concept.

This allows the operations team to lower the volume of log traffic by eliminating the DEBUG and lower messages, and set up alerts on the WARNING and higher messaages.

OpenTracing

It's great that you have a session ID to track user behavior. Including that information in your logs helps track things at a session level. Going a step further than that, many cloud log systems have support for something called Application Performance Monitoring (APM). And one of the most commonly supported standards to support that is OpenTracing.

This allows you to trace a specific request from start to finish, even across asynchronous communications. There's work to be done to define where each context begins and ends, but OpenTracing provides a way to track child contexts with parent contexts. That way the logging aggregator you use can visualize that for you in a dashboard, and you can drill in to the specific log messages that apply.

It's particularly useful for the following reasons:

  • You can calculate the overall processing time of every action
  • You can detect outliers where the performance is way out of normal
  • You can drill in and find out what happened in the outlier

Conclusion

Make sure you know how you are going to consume your logging data, and design for that use case. Know where you can decorate your logs outside of your application so you can divide the responsibilities appropriately. And think about what is necessary to support the application once it's deployed.

Your first step into the process may not be 100% correct, and that's OK. If the app is successful, you'll have the opportunity to add in log messages that are needed, and remove or downgrade the level of log messages that are no longer useful.

2
  • I feel very uncomfortable criticizing such extensive answer but to me it sounds a bit useless, I am sorry. I was not asking about logs consumptions, I said that I am going to use STDOUT and Loki. Yes, I can use some dispatcher like Fluentd outside of the app to filter logs but the question was more about intra-process logging practices. Feb 2 at 18:04
  • Logging without considering consumption is going to leave you with logs of dubious usage. Just some experience speaking here. Feb 3 at 11:58
0

It sounds like you have at least 2 different types of logs for two different audiences:

  1. "backend engineers" (developers?)
  2. "user support and analysts"

Many orgs also have a third:

  1. Ops

What to log for a given audience is independent of severity. So, to do a great job of this you may need a way of tagging a log entry such that the intended audience(s) are identified. There may also be spots in the code where different log entries are written for different audiences.

The mechanics of getting log entries into the right logs is the easy part. Good luck!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.