I'm writing a log parser that continuously looks for specific patterns and then fires off events to another system when it finds these matches. The firing of the event is time critical to catch errors or ensuing problems. This part is non-negotiable in terms of design.

The log files belong to Applications and the Operating System; thus they can be rapid in logging, large in size and rotate based on time, or size.

What is a better approach to reading these log files, detecting when they change, and reading from their last read location as I wrote code to do this using a RandomAccessFile which kept the last read location for every file.

For dealing with large files and rapid logging I did this:

  1. DataProducer - would get last read point in file and then find any changes then add to a Queue
  2. DataConsumer - any changes to the Queue by way of size, would then cause this thread to wake and start parsing all the items in the Queue for that log file. Deleting what it has parsed.

I am looking for some suggestions (in light of what I have done) that may improve the current way I manage reading a file, and picking up when there are changes, and parsing the file efficiently (considering size and rotation). Code or an API will do.


Sounds like a job for Splunk

You probably don't want to re-invent the wheel here. Splunk has a free option and can provide you a lot of extra useful functionality that you will probably need further down the road.

  • thanks for the suggestion. But would prefer to write it to fit inline with our in-house technology suite :) – Dane Balia Jan 21 '13 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.