Let me start by setting up the context briefly.

In our organization (a Big Data company), there are many different systems, including Web sites, workers (system listening to queues and/or topics), scheduled processes (processes triggered ), and so on written if different technologies such as .Net, Java, Python, etc.

As the organization grows, so do the microservices ecosystem and the amount of data involved. Most of our systems write logs to local files, but since some of them are quite older than others there is no unified approach. Since we don't have a well-defined logging architecture and most of our systems write logs to local files, taking advantage of those logs has become difficult. We are not able to react proactively, and read the logs is complicated, often useless.

We have identified these requirements:

  • There must a way to group and track all related logs
  • Logs should be easy to read and query
  • Performance must not be degradated
  • Each log should contain:
    • Timestamp
    • System
    • Environment
    • Instance
    • ActivityId (Grouping factor)
    • Class + Method
    • Useful information
      • If it is an exception, call stack, message, line, input, etc.

And based on those requirements, we came up with the following architecture:

Architecture Overview

A pseudo-implementation should look like approximately like this:

When a system receives a call, checks if that calls already includes an ActivityId, if not creates a unique ActivityId. Each Log then will contain that ActivityId, and every subsequent call to other systems will include that ActivityId

The logging component must smoothly send the logs (in batches or one by one) to the streaming service.

The questions that arise are:

  • Are we missing something? Does the architecture make sense?
  • What kind of considerations should we take into account?
  • Are there some components that should be present and that actually are not?
  • Regardless of implementation mistakes, as an architecture, would it work?

There are some other questions here, but most of them refer to implementation more than to an architecture.

Since we are still in the design phase, we haven't gone deeper into implementation details, but we have seen some good approaches using Serilog and Dataflow for .Net and Log4J for Java.

Any recommendation or suggestions are welcome.

  • Seems like a lot of things to go wrong.
    – Telastyn
    Aug 4, 2018 at 22:44
  • Might I ask what you mean? What do you see like a point of failure? I would appreciate you letting me know in case you downvoted the question. Thanks. Aug 5, 2018 at 4:38
  • use log stash to pick up and parse your existing log files
    – Ewan
    Aug 5, 2018 at 8:10
  • I didn’t downvote, this is an okay question imo. Each line in a diagram is a potential point of failure. Since you’re logging a message, something in the system has already gone wrong. I’d worry that such a complex logging system would not be resilient to things being broken. Either you don’t get the log messages that you need to diagnose stuff, or worse the log system blows up and takes the rest of the system with it.
    – Telastyn
    Aug 5, 2018 at 14:13

1 Answer 1


Some thoughts.

The picture does not show machine boundaries. If you care about performance you may want to have a service log to a local file, possibly in its own format, and have some other local program forward the log entries to a central database in a lazy manner, when things have cooled down.

My experience with logging is that a lot of text is produced and no one cares about it until they're really stuck. The more you log the less attractive it is to wade through that log file, you typically want to produce one scenario and would replay it anyway so the value of history will be limited (we are not talking about journals, trails, transaction logging). Most of the time it would be a waste of bandwidth and storage space. So you want something to filter on the production side, like a severity level (debug, info, warning, error, fatal) that you can set at startup (or maybe even at runtime), just before you start analysing, for those services that need attention.

To make this work/useful you need all your existing services to follow the same logical logging policy. That could be a lot of work, like reviewing and reworking all existing code. This sounds like a big project in its own right and it sounds hard to sell.

  • I know that most popular logging libraries require the developer to choose a severity level when logging messages, but this is an anti-pattern because there is no way that most classes that log issues can know how severe this issue is to the system as a whole.
    – bikeman868
    Aug 5, 2018 at 8:38
  • Hmmm... How is that? An unhandled exception is fatal. An unexpected exception is an error. An expected exception is a warning. Anything that helps tracking regular activity is info. Small steps within an activity may be signaled with debug messages. In each case I know perfectly well what the severity level should be. And I know which ones I would be interested in in a particular debugging scenario. Aug 5, 2018 at 11:27
  • This is kind of a big subject to discuss in comments, but anyway, think about these two aspects: as a consultant brought in to sort out messes I have seen so many systems where the logging is completely useless mostly because these severity levels were abused by the developers (for good reasons); the severity should be defined by the consumer of the logs not the originator of the log messages, ie alert me if this type of error happens in this module and it is not part of the module's single responsibility to know these rules.
    – bikeman868
    Aug 5, 2018 at 19:05
  • A developer can determine the severity of a situation. But defining the actions to take when a specific situation occurs is something else. Instead of building a completely new central logging system I would look at logstash first, like @ewan commented.
    – Kwebble
    Aug 5, 2018 at 22:19
  • @MartinMaat Thanks for your answer. One of the problems we actually have is that some of our systems receive millions of requests per minute (The ingestor for example), and sometimes errors in these systems are not shown until 2 days later. Even though, as you have mentioned, most of the time logs are useless, in some cases is rather difficult to reproduce cases. I take your advice, probably we don't need to use the same logging mechanism everywhere, but in the core. Aug 6, 2018 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.