We've got 2 very large platforms for our services & jobs.

Both platforms consists of 20+ servers hosting 1000+ services/jobs.

Each job/service is essentially a java web application.

Both platforms are managed centrally, which allows us to get a holistic overview of the platform, as well as drill down capability to each server.

Each server has the following logs:

  1. System out log
  2. System err log

The service/job log is centralized and not stored on the server.

A service/job application may include 3. party libraries that write out to sys out/err, which makes our operations difficult, as the centralized log is not the solemn source for troubleshooting.

Architecturally we only want the sys out/err log to contain server log entries, not service/job log entries. This would allow the development department to troubleshoot business logic via the centralized log and the operations department to troubleshoot sick servers via server logs.

As we have no control over these 3rd party libraries, and the sys out/err is global to all the applications on the server, how do we best approach this challenge?

  • I would look into some a log management tool such as Splunk to manage all of your many different types of logs and log sources and allow all kinds of filtering/querying. A lot may be doable with the tool straight out-of-the box, other bits may need some extra work or configuration to parse the log data efficienty, but these kinds of tools generally have quite a lot of different options. (i.e. rather than needing to adapt your system and architecture, you can probably adapt the tool) – Ben Cottrell May 25 '20 at 15:44
  • 1
    can you give an example on how splunk can help with the above scenario? – Kelvin Wayne May 26 '20 at 6:25
  • I can't speak for exact implementation details, but systems I work on include hundreds of different services writing logs in various formats. Splunk has a lot of very sophisticated viewing, filtering, querying and parsing/processing capabilities, so someone consuming the log data can include/exclude whatever they do and don't want to see. For example, I can quickly switch between different types of logs from IIS logs to Application logs in various different JSON and XML formats, as well as some which are just plain "lumps" of text. – Ben Cottrell May 27 '20 at 14:23
  • The logs themselves can be assigned to different indexes for searching/grouping, but can also have different tags associated with them for filtering, as well as a lot of built-in options like the hostname, the logfile name (or an identifier for the log source if not a file). splunk itself can also have custom parsing rules to generate custom tags too - for example, a regex pattern could be used to identify something in a log entry and tag it accordingly. – Ben Cottrell May 27 '20 at 14:23
  • This is my personal experience as someone who is a user of splunk rather than someone who has had to set it up from scratch, I'm not an expert in it by any means. I could not speak for the effort required (I assume there's some work to customise and configure), just that it seems to have a lot of very useful capabilities for only being able to filter out the vast majority of "noise" and (mostly) only see what you want to see! – Ben Cottrell May 27 '20 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.