What should I do with production error logs?

Question

We have a PHP intranet site that we run with warning/error logging on (as it's not under heavy load and there's a number of bugs/bad code left to find). Because the site is relatively unstable and in active development, I've been clearing the production error logs when each new version releases so I get a clean look at what problems we're having. We usually don't release until all logged problems identified in the last production build are fixed.

I still keep Apache's error logs because our code shouldn't mess up Apache (and if it does it's a huge deal), so if something's wrong there I want to know.

I'm wondering what a good practice is here; our previous code base logged hysterically verbosely so I've been trying to limit the logs so it's actually usable information, but just clearing the logs seems suspect as well.

For reference, most of the logs are undefined variables because a past programmer liked to use if (!$var) instead of if (empty($var)). I've been eager to remove those entries since the application doesn't perform any unexpected actions but I would get dozens of the same warning daily until we fixed it. Those are also why I left warning logging on in production.

Should I keep all my error logs for future reference? Should I segregate old logs somehow and tag them to build numbers?

Answer 1

If you are using them to eliminate the bugs being reported, then they are doing their job.

They are useful NOW in the identification of current bugs, but if they are all fixed then I'm not sure there is much value in keeping the old logs once issues from them have been dealt with.

Answer 2

I'd save them and stick them in a zip file or some other archive. Archiving should compress them, so they won't take a log of space, and they'll still be easily accessible if you want/need to review them. Name the files after the release/deployment ID so it's easy to correlate them with the version of the software that produced them.

Answer 3

Logs are a useful source of information. Though a lot verbose, they are really good for researching and zeroing in on a problem. My suggestion would be

1. Use a bug-tracking system to track frequent errors. Trac is a good way to start. Though the error logs are large, it would be caused only be a few bugs. A bug-tracking system would help you to track bugs in a compact manner.
2. Archive your error logs in a different database. SQLite is a good choice. If your error logs are extremely big, I prefer a key-value database such as Redis or a hierarchical database such as HDF5. You can always use queries to extract data
3. Track the bugs with your error log report for better control. Assuming a page not found error has been hit 100 times and the bug being the page misspelled, you have Bug1 occurrence 100 times. This would make it accessible to all so that errors can be quickly ironed out. Though its a bit difficult, its a better long-term solution.