Let me start by setting up the context briefly.
In our organization (a Big Data company), there are many different systems, including Web sites, workers (system listening to queues and/or topics), scheduled processes (processes triggered ), and so on written if different technologies such as .Net, Java, Python, etc.
As the organization grows, so do the microservices ecosystem and the amount of data involved. Most of our systems write logs to local files, but since some of them are quite older than others there is no unified approach. Since we don't have a well-defined logging architecture and most of our systems write logs to local files, taking advantage of those logs has become difficult. We are not able to react proactively, and read the logs is complicated, often useless.
We have identified these requirements:
- There must a way to group and track all related logs
- Logs should be easy to read and query
- Performance must not be degradated
- Each log should contain:
- ActivityId (Grouping factor)
- Class + Method
- Useful information
- If it is an exception, call stack, message, line, input, etc.
And based on those requirements, we came up with the following architecture:
A pseudo-implementation should look like approximately like this:
When a system receives a call, checks if that calls already includes an ActivityId, if not creates a unique ActivityId. Each Log then will contain that ActivityId, and every subsequent call to other systems will include that ActivityId
The logging component must smoothly send the logs (in batches or one by one) to the streaming service.
The questions that arise are:
- Are we missing something? Does the architecture make sense?
- What kind of considerations should we take into account?
- Are there some components that should be present and that actually are not?
- Regardless of implementation mistakes, as an architecture, would it work?
There are some other questions here, but most of them refer to implementation more than to an architecture.
Since we are still in the design phase, we haven't gone deeper into implementation details, but we have seen some good approaches using Serilog and Dataflow for .Net and Log4J for Java.
Any recommendation or suggestions are welcome.