I'm currently analyzing a problem in a old "distributed" system, that only occurs sporadically - as the system does not support other means of debugging help I have to rely on the logs (I did not design or develop that system).
Currently I do only the steps to trigger that error (of cause only sometimes, without any obvious correlation).
What I'm now thinking of is to find something in the Log file. the manual analyzes did not show me anything so I was thinking about support of a tool ...
What I'm thinking of is scanning all log files of all components and than filtering the entries of the logs for entry correlations that only can be found in the time slots where the error did occur.
Maybe with that information it would be possible to find an hint to find the underlying problem (I guess on some kind of race condition like).
The Logfiles in my concrete case of the different components are log4j (and an Apache Proxy).
Is there any information on some approaches like that? Or is there even a tool (program, library, ...) supporting on that case of analysis?
Or Would I have to code it myself - if so is there some kind of algorithm described for that kind of problem?
What I currently have in mind is:
- read in all log files
- convert to common format (date-time, data)
- filter for that time slots where the error occurred and compare the correlation between entries that only occurred in that time slots compared to other time ranges to find some correlation functions that have a maximum only in the suspicious time slots.
Unfortunately I don't know how to do ...