Logging Architecture Solution

Question

Let me start by setting up the context briefly.

In our organization (a Big Data company), there are many different systems, including Web sites, workers (system listening to queues and/or topics), scheduled processes (processes triggered ), and so on written if different technologies such as .Net, Java, Python, etc.

As the organization grows, so do the microservices ecosystem and the amount of data involved. Most of our systems write logs to local files, but since some of them are quite older than others there is no unified approach. Since we don't have a well-defined logging architecture and most of our systems write logs to local files, taking advantage of those logs has become difficult. We are not able to react proactively, and read the logs is complicated, often useless.

We have identified these requirements:

• There must a way to group and track all related logs
• Logs should be easy to read and query
• Performance must not be degradated
• Each log should contain:
  • Timestamp
  • System
  • Environment
  • Instance
  • ActivityId (Grouping factor)
  • Class + Method
  • Useful information
    • If it is an exception, call stack, message, line, input, etc.

And based on those requirements, we came up with the following architecture:

A pseudo-implementation should look like approximately like this:

When a system receives a call, checks if that calls already includes an ActivityId, if not creates a unique ActivityId. Each Log then will contain that ActivityId, and every subsequent call to other systems will include that ActivityId

The logging component must smoothly send the logs (in batches or one by one) to the streaming service.

The questions that arise are:

• Are we missing something? Does the architecture make sense?
• What kind of considerations should we take into account?
• Are there some components that should be present and that actually are not?
• Regardless of implementation mistakes, as an architecture, would it work?

There are some other questions here, but most of them refer to implementation more than to an architecture.

Since we are still in the design phase, we haven't gone deeper into implementation details, but we have seen some good approaches using Serilog and Dataflow for .Net and Log4J for Java.

Any recommendation or suggestions are welcome.

Answer 1

Some thoughts.

The picture does not show machine boundaries. If you care about performance you may want to have a service log to a local file, possibly in its own format, and have some other local program forward the log entries to a central database in a lazy manner, when things have cooled down.

My experience with logging is that a lot of text is produced and no one cares about it until they're really stuck. The more you log the less attractive it is to wade through that log file, you typically want to produce one scenario and would replay it anyway so the value of history will be limited (we are not talking about journals, trails, transaction logging). Most of the time it would be a waste of bandwidth and storage space. So you want something to filter on the production side, like a severity level (debug, info, warning, error, fatal) that you can set at startup (or maybe even at runtime), just before you start analysing, for those services that need attention.

To make this work/useful you need all your existing services to follow the same logical logging policy. That could be a lot of work, like reviewing and reworking all existing code. This sounds like a big project in its own right and it sounds hard to sell.