How do you test software that is time sensitive?

Question

By time sensitive I mean for example a script that only runs once a month or a script that runs continuously but gives a certain output only once a month. Obviously you can unit test for a lot of cases but there are exceptions ( in my understanding).

A recent example I ran into was setting up a cron job to run on the second to last day of every month. This required using a shell script with cron tab to get the correct day of the month for cron, something like:

1 0 [shell command] * * [my script]

I was unfamiliar with the script and unfamiliar with shell scripts in general and so had no good way to test it other than waiting for the end of the month to come and seeing if the script executed correctly (actually my solution was to find a co-worker who knew a lot more about cron and shell scripting that I did).

So I am curious if there are any useful work arounds for testing time sensitive scripts.

Answer 1

In addition to unit testing, there are two other strategies for setting up automated tests to deal with an OS-specific issue:

• Virtualization: You set up several OS images (for example, using VMWare) with the exact configurations you require, set up a way to automatically pull the binary to test (usually by mounting a special directory into the VM's space), and then execute the test.

Or:

• Instrumentation: Manually add special if conditions to your program that will make the program behave differently. Under Unix, this would be done by checking if a certain environment variable is set, like FOOBAR_TEST_TIME_WITH_T=500. Your automated tests will then just use different settings of the environment variables, and different environment variables, to execute what you need.

You can also link to different libraries, if your interactions can be expressed at the library level, which you can think of as a virtualization (if the "library" is the OS kernel) or as an instrumentation technique. Both terms may be used, although the term virtualization as-used today almost always means something like VMWare. A library specifically for returning canned values or re-running specific interactions would be a mock or stub approach.

There are also automatic instrumentation tools, which can rewrite your binaries to get other desired effects, like the file system being full.

Overall, your goal is to find bugs. To check for weird cases like the file system being overfull, it's easiest and still effective to just manually instrument your program, going the virtualization or manual machine configuration route rarely if ever.

Answer 2

The most effective - Change the date of the machine that you are testing on. Set it for a little before when it needs to run, and verify when it kicks off and that it runs correctly. This is not always possible however if there are multiple machines involved, or, if resources are required that your company has no control over. Make sure that you're doing it for multiple months however, and make sure that you change the year a couple of times to test February.

Answer 3

I don't know about your script, but I try to use some sort of parameter where I can set the date. In your case, if no date is given, default to end of month. In your code, take the date parameter and run if today is two days before. Not only will you be able to test it (pass in a date two days from now), but also run the next day in case something prevented it from running under normal circumstances (power failure, server down, etc.).

Answer 4

Since you mentioned crontab, I assume you are running in a nix environment. In that case, I think it would be worth your while to check out libfaketime:
http://www.code-wizards.com/projects/libfaketime/

Through the magic of LD_PRELOAD, we can load custom versions of library functions as long as they match the interface. What libfaketime does is load versions of the time system calls that allow you to customize their behavior via environment variables. You can force time() to return a hard-coded value or an offset from the current time, all without effecting anyone else on the box.

Answer 5

There's not a lot of need to test cron, as it's pretty much been tested ("production test") for many generations. Of course, if you're working with a shell script, you might set the date/time in a virtual machine.

The preferred way to deal with this is "mocking the clock", using one programming trick or another to fake the time. In shell scripts you can use the ${:-} syntax to use a date set in an environment variable, and fall back on the actual time if it's not been forced.

In other languages we use mock libraries or build an abstraction over the clock.

The mock clock is good because you can automate it instead of having to manually set up your test. This is a very great benefit when it comes to modifying the script/code later and you can easily tell whether it still works or not.