We've got 2 very large platforms for our services & jobs.
Both platforms consists of 20+ servers hosting 1000+ services/jobs.
Each job/service is essentially a java web application.
Both platforms are managed centrally, which allows us to get a holistic overview of the platform, as well as drill down capability to each server.
Each server has the following logs:
- System out log
- System err log
The service/job log is centralized and not stored on the server.
A service/job application may include 3. party libraries that write out to sys out/err, which makes our operations difficult, as the centralized log is not the solemn source for troubleshooting.
Architecturally we only want the sys out/err log to contain server log entries, not service/job log entries. This would allow the development department to troubleshoot business logic via the centralized log and the operations department to troubleshoot sick servers via server logs.
As we have no control over these 3rd party libraries, and the sys out/err is global to all the applications on the server, how do we best approach this challenge?