9

I understand that log file rotation is changing the log file you used when (1) one gets big enough or (2) at EOD, but I'm not sure I understand the reason for (1). I have never had any issues with large files and cannot think of reasons why we might set an arbitrary limit (for example, the limit we were given was 10 MiB).

20
  • 39
    You've never had to consider a service that might fill up a disk with log files?
    – pjc50
    Apr 24 at 8:50
  • 21
    no one is mentioning being able to delete logs older than X easily?
    – Ewan
    Apr 24 at 9:54
  • 5
    I have encountered several log files that Notepad++ couldn't or struggled with opening. Yes, there are dedicated tools to read line-by-line, but it's a pain to use. And an unnecessary one
    – Hobbamok
    Apr 24 at 14:15
  • 5
    Another reason to rotate is application restart: this way it's easy to find the start/stop of the application, and it's clear (by number of small files) when it's restarting like crazy. It also makes it easy to learn why it's restarting: jump to the end, here you are. Apr 24 at 17:08
  • 10
    How does reducing the size of individual logs change the fact that disk will fill up by deleting old files. The whole point of separating them into files in the first place is to be able to easily select old logs to delete. Deleting lines from files is a lot more difficult than deleting files from folders.
    – slebetman
    Apr 25 at 2:26

8 Answers 8

59

"I have never had any issues with large files" is not synonymous with "there is never a problem with large files". Your lack of experience with problems does not prove that there aren't any, it just proves that you haven't experienced them if they exist.
Anecdotally, I've had to work with >2GB text-based log files, and this was back in the x86 era where RAM was limited to 4.2GB (including your OS). Large files become effectively impossible to handle, making them useless as a troubleshooting tool.

The short and sweet answer here is that performance is impacted by file size once you get to non-trivial file sizes. As a very simple example, how are you going to easily look through a file whose size is larger than the RAM of your machine?

10MB is a very low cutoff point, but I suspect that this limit hay have been chosen partly in relation to the physical size of the file in terms of manually scrolling through it and reading the logs. We can argue about what a more sensical cutoff would be, but it should be clear that some cutoff based on file size makes sense, so as to avoid the problems that arise when files get too big.

13
  • 9
    This is really an issue of tooling. It is absolutely possible to work with (view, search through, even edit) files thar are much, much larger than physical RAM. You just need tools that are explicitly design to handle them, e.g. liquid-technologies.com/large-file-editor Apr 24 at 8:33
  • 28
    @MichaelBorgwardt yeah but requiring a specialized tool just to glance at /search through log files isn't ideal. Not to mention that trimming that log becomes equally tedious
    – Hobbamok
    Apr 24 at 14:16
  • 18
    @MichaelBorgwardt It's not impossible for me to buy a wheelbarrow to get all my groceries from the car to the kitchen in one go, but it's decidedly easier to take a second trip instead.
    – Flater
    Apr 25 at 1:33
  • 12
    @Basilevs grep is over 50 years old now, it comes from a time where memory was very scarce. I would say it is specialized in handling files that don't fit into ram, even if it was probably implemented like that out of necessity.
    – kapex
    Apr 25 at 9:55
  • 3
    Just to summarize the ongoing comments: we can always argue about different solutions if you're willing to put in more effort, learn different things, or accept different limitations of your approach. The question here wasn't "name alternatives to log file rotation", it was "what is the advantage of log file rotation", which shouldn't devolve into listing every other possible solution to this problem under the sun.
    – Flater
    Apr 25 at 23:32
29

Log file rotation is not only related to technical issues with the file itself. It's also related with operational considerations, like:

  • being able to incrementally backup the logs (especially in case of transaction logs)
  • reducing to a decent minimum the risk of loss of events logged (open files during a hard system crash can be at risk)
  • passing log files to a secure archival for preventing tampering, or to a SIEM system for threat detection.

While it is less problematic nowadays to deal with large files, large files (constantly reopened) negatively affect these needs.

If you only log debugging information, you can ignore these constraints. but for a long living large system , especially with binary logs, the question is not IF you'll face a byte corruption but WHEN.

8
  • 1
    True, it also depends on the level and detail of what you're logging. Just startup times and errors? 10mb will suffice. Are we logging every user input or network interaction? Things will get heavy quick
    – Hobbamok
    Apr 24 at 14:18
  • I am only asking about (1) vs (2) like I mentioned. Why not rotate daily instead of both daily AND based on a size limit? Apr 26 at 3:24
  • You need to design your system as best supports its purpose and not blindly follow broad recommendations. So it all depends on the purpose of your logs and what you want to protect. If you use an oracle DB for example, the logs are transaction logs. I think they first came up with a recommended size of 10 MB or the next shutdown (no end of day in a 365/24 system). These logs allow to restore the DB in case of crash and replay up to the latest committed transaction. If you have no size limit and wait for end of day to rotate, you can not withon the day backup the log. If the disk crashes...
    – Christophe
    Apr 26 at 6:06
  • ... you then have lost a day of business transaction. A catastrophy. If it's an audit log, a hacker may come in, do some fraudulent activities, eventually manage to find the log and erase the traces there. If you have a size limit (that in practice can be converted in time limit given a known throughput), you can reduce the risks by periodically rotating the logs and sending logs to a secure storage to protect them, or to a siem that could detect before the end of the day that something fishy is going on. Of course don't use hard coded limits but allow admin to configure the desired size ;-)
    – Christophe
    Apr 26 at 6:11
  • Finally, if it's for debugging purpose, you can very well do as you say. In the worst case you would have to be patient for the next time the bug occurs. So it's all about the question of what loss you can afford
    – Christophe
    Apr 26 at 6:12
15

The other answers already provided very useful reasons, but there is one more I'm surprised I haven't seen outside of a comment yet:

To be able to delete old log files to free disk space

A very busy application logging all its processes will do a lot of logging. A month's worth of logs might be hundreds of megabytes.

If you close your log file when it hits a certain size or at end of day, your log folder will contain many files, each with a date when it was written. When your administrator gets an automated warning that the disk is starting to run out of space, he can simply go into the log folder and delete (or move to some cheaper archive) everything that's from last year or older. Hit delete and enter, or ctrl-X and ctrl-V, done.

If you just keep writing into one master log file, then even if none of the problems mentioned in the other answers apply (and they will) your administrator would have to follow a more complicated process when disk space runs out. Best case, he'll rename the logfile (hoping that no write action overlaps exactly that moment), your application will see "no file with the right name" and create a new log, then he can... hmm.

Can he delete the old logfile? No, it contains logs up to just before he renamed it. It might still be needed for days, weeks or months!

Can he move it to archive? He can, but then he'll have to make sure any developer who has to look at the logs has access to that.

Most likely, he'll have to rename it, wait a few months (during which time hopefully the disk does not fill up completely), and THEN move it. Two processes instead of one.

Time rotation: to make it easier to find the right log file for an incident at a known time

Rotating your log files every month/day/whatever also helps you find the right log file to read if you have to investigate an incident. A user reports "the application crashed on 2024-04-13 at around 11 am" and you can just go to the logfile that is named "app-log-2024-04-13" (or similar) instead of having to sort all your log files by creation date and scrolling until you find the right one

3
  • Such administrator would end up doing log file rotation manually. :-)
    – Pablo H
    Apr 25 at 14:21
  • 1
    Ok, but this doesn't answer my question about why both the size and time rotation are needed. That is specifically what my question is about. Apr 26 at 3:24
  • @user129393192 without the size rotation, you might end up with one huge logfile for the month/day/whatever you set your time rotation to, leading to all the size-related issues that mostly the other answers cover. Without the time rotation, you get at least some of the issues described here... oh, and one I will edit into my answer: if you have to find the log for an incident at <date and time>, it's easier to find the right file out of hundreds if you know that each file covers exactly one day, and is named for it.
    – Syndic
    Apr 26 at 5:50
8

never had any issues with large files

and cannot think of reasons why we might set an arbitrary limit

Every system I have ever ssh'd into has had limited budget and limited attached storage. If you run $ yes > big.txt you'll eventually see "FS full", and similarly for a busy syslogd.

Often we value uptime, and are loathe to see app errors caused by a full partition. An app might typically log messages at a limited rate, say an average of 10 line / second. Sometimes the internet clients stimulating the SUT will change their behavior, so we see 100 line / sec. Or the stimulus is ordinary but a local "verbosity" config change makes us log at an unusually rapid rate.

Given a 10x increase, a daily log rolling policy would be slow to respond, and by end-of-month the steady state size of /var/log would expand to ten times its usual size. OTOH, a size-oriented policy would respond immediately, and keep the total size of /var/log essentially constant (± 10 MiB in your example).

The tradeoff, of course, is we'd have just one-tenth the forensic evidence available to investigate the anomaly, so for a delayed investigation it's likely we wouldn't see events near the transition point, as they have already been aged out and deleted. C'est la vie!

Many people would be willing to accept that tradeoff, if it means never hearing "My daemon died when it failed to allocate space for a temp file!"

9
  • I think there's a mistaken assumption here: log file rotation only means creating a new file, it does not imply anything about what happens to the (now closed) files, and by default they'd just remain there and still occupy space. On the other hand, once you have log file rotation, a simple crontab can be used to delete all but the last N log files for a given application which will help keep things under control. Apr 24 at 17:06
  • 8
    @MatthieuM. I've always considered file removal as part of 'rotation'. Logging tools like Log4J, for example, let you set the max number of files to keep. To me that's what makes it 'rotation' instead of just 'splitting'.
    – JimmyJames
    Apr 24 at 17:58
  • @MatthieuM. The OP was asking about (1.) size based versus (2.) calendar time based logrotate policies. I explained that, if uptime matters and there's risk of filling a partition, a size based policy gives us a better chance of maintaining 24x7 uptime, with smaller variance in the output of du -sk /var/log. Asking syslogd every now and again to start a new file enables space management policies; it affects the size of the resulting files and the granularity at which we can reclaim storage space. Log4J and others take a similar approach.
    – J_H
    Apr 24 at 18:11
  • @JimmyJames: I guess our experiences differ. Coming from the C++ world, there's no such thing as standards, and the one (custom) implementation of log "rotation" I saw was just splitting files -- because large files are unwieldy to open -- and nothing else. Apr 25 at 7:28
  • I think the word "rotation" comes from the idea of reusing the oldest file. If log files have fixed size, and their number is fixed/capped, this gurantees always having space, avoids (rotating) disk fragmentation, and perhaps has slightly better performance because of filesystem caching (though for logging it would probably be deactivated).
    – Pablo H
    Apr 25 at 14:19
6

I worked on a bug last year where an action in production was suddenly failing due to a timeout. Fast, near instantaneous in dev and QA, failing in production. Took weeks of on again off again work before someone with the right rights noticed the huge log file. Writing to the log file was the source of the problem.

Now, you might say that was just a bad way to write to the log, but it’s a fact that log rolling (which was available, just not turned on) would have saved us a lot time.

Just because it’s never been a problem for you, doesn’t mean it’s not a problem.

1
  • I didn't ask about log rolling in general. I'm asking about log rolling both based on time (once daily) and ALSO a size limit, as opposed to just one. Apr 26 at 3:26
3

Limit total amount of resources on system

In addition to some of the other points mentioned, I would like to highlight that some systems take care to limit the total number of resources. Not only disk space, also memory, database and other aspects.

Examples may include large scale critical applications for controlling factories, power plants, electricity grids, but also the opposite end, for instance a computer in your car.

Everything that should be able to run years without intervention and should never fail for some weird reason.

Using log file rotation is one of several things to consider if one aims to achieve the above.

Obviously on large scale systems this is now less an issue than some years (or decades) ago, but one may still want to consider it.

2

The original reason for using size as the rotation criteria almost certainly came from the limitations of 32-bit operating systems, where most applications (and maybe the OS itself) couldn't handle files larger than 2GB. So it's necessary to rotate log files before they reach this size. Even if the system where the logging is done can handle larger files, you may want to limit them in case there's a need to copy the logfile to a system that can't (e.g. for archiving, or analysis).

And even when there's no hard technical limit to file size, working with extremely large files can be inconvenient. Many file operations that you're likely to do with log files are O(n): E.g. loading the file into a text editor or searching for messages matching a pattern or containing a particular string.

There's little downside to limiting logfiles to a reasonable size, and it usually makes things easier, so it's a common practice.

-3

Hopefully, if using the Twelve-Factor application design, you would be writing logs directly to stdout. Let the log collection agent (that is either attached as a sidecar on a containerized application, or just another running background process for a standard server-side application) handle the gathering of logs from all the process's stdout streams and the forwarding of logs to a centralized logging solution.

If this is not possible for your application (you don't have a log forwarding solution in place), log rotation is useful for general filesystem usage maintenance. Sure, your application right now might not be generating a lot of logs, but imagine a much larger application that processes hundreds of requests per second, all being logged with their connection metadata and the application flow for each transaction. You can imagine how large a single log file can get in this instance (especially with DEBUG logs turned on), and therefore it needs to be rotated.

The exact threshold for rotating a log file depends on the application. If an arbitrary rotation threshold is X (can be measured in bytes or simply lines of logs), and your application only generates enough logs to reach X in a week, then you can rotate your logs weekly. In turn, if your application reaches X in a day, rotate daily - an hour, rotate hourly, etc.

Its up to you and your team to determine what makes sense as your logging threshold X in order to help you debug an issue. Obviously, the larger your threshold, the more lines you would have to search through when debugging some logged transaction, but in general, document search utilities such as grep should be just fine in helping to debug an issue for a given time period under threshold X.

TL;DR: log directly to stdout if you can, assuming you have a log forwarding agent installed that is reading your application's stream. If that is not possible, determine some threshold X (e.g. number of log lines) that would be the maximum size of a log file you would be comfortable sifting through while debugging an issue, and rotate your logs at the general time interval that they reach X.

13
  • 5
    I'm not the downvoter, but there are many things that can go wrong with stdout. What if the application crashes? What happens to stdout then? What if the application creates logs faster than the agent can consume? In dev scenarios, a terminal only log seens pretty terrible, most terminals have really poor ux capabilities.
    – Ccm
    Apr 24 at 6:36
  • 19
    Logging to stdout is also somewhat orthogonal to the question. The logs have to go somewhere and you still have to think about how long to keep what (and at what point to dump to files in archival storage - and how long to keep those too). Logging to stdout is an interesting one. I'd rather dump output that is more human friendly to the console and well structured json - with more detail - to somewhere else. Everything to stdout means you have to compromise. (Like config with environment variables - its good in theory, but not necessarily ideal in practise.)
    – Murph
    Apr 24 at 8:28
  • 6
    Mixing logging into your output stream is crazy. How is any downstream process going to separate logging from actual output? Logging to stderr may sometimes make sense, but even so.... Apr 24 at 14:32
  • 6
    @TobySpeight the philosophy seems to be more for things like eg. web apps running in a Docker container, where you normally wouldn't use the stdout for anything functional, and it's easy to capture by some sort of sidecar or hypervisor. Not for a command-line utility that might actually want to use the output for, well, output. Apr 24 at 14:50
  • 10
    I didn't downvote, but preaching someone's random 12 commandments as gospel that people are "hopefully following" does irk me a little.
    – marcelm
    Apr 24 at 19:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.