What's the best way to avoid catastrophe caused by negligence? [duplicate]

Question

I have been a programmer for almost 1 year.

As an ADHD adult, naturally I don't have the same strength of attention on ordinary stuffs as my colleagues do.

And I find the catastrophe made by me are usually caused by trivial negligence.

Like for today, I found the cron process on the server collapsed in the morning.

After half hour of debugging. I found I wrote in the cron

* 4 * * * sh daily_task.sh

instead of

0 4 * * * sh daily_task.sh

Which runs the huge shell 59 times in the morning instead of the intended 1 time.

Is there some kind of cultivatable behaviour or some tools or anything that can help me at least reduce such kind of mistake?

How do you do to avoid such kind of mistake?

Answer 1

Is there some kind of cultivatable behaviour [...] that can help me at least reduce such kind of mistake

Absolutely, it is called four-eyes-principle.

If you had you shown your crontab entry to a second person (a person knowing cron, of course), chances are high the mistake would have been avoided.

In programming, when it comes to this, people mostly think of code-reviews, but that is actually the same thing.

Answer 2

You don't need ADHD to have this problem. It might be that you are working in environment where you get interrupted often or where you have to work on many things at the same time. So loosing track of your progress or making small mistakes is almost guaranteed.

There is one thing you can do to reduce those errors : tests based on specification.

First, you write down the specification in the most detailed way you can. In case of code, this should be in form of automated tests. Best way is to follow TDD. So when you suddenly get distracted, then your tests will tell you what you have already done and allow you to quickly pick up where you ended. Also, it makes regression testing easier.

In a case, where testing code is not possible, make sure you have someone else to go over what you have created and have them test the software according to the specification. You should always test software in testing environment before it goes into production. The issue you are talking about would be easily caught by running it for a day in testing environment and checking the logs.

Answer 3

You always have to be at least two steps away from disaster. That means you never directly push something you just wrote into production. You either test it, or you have it tested, or you have it reviewed, or figure out some other way to have a two step path to disaster. Sometimes I will leave something for the next day and review my work again before committing.

In cases where something is really critical, you add a third step: a second reviewer, a second independent test, etc...

To err is human. Our brains are not built for precision. Everyone makes mistakes, everyone writes bugs, although some more than others (being careful still pays). You have to figure out a process that builds in the assumption that any individual's work is flawed.

Answer 4

Previous answers mentioned two points:

• Reviews.

• Testing.

but there are also three additional elements which may help:

• Better visualization.

  By using tools which let you better visualize what you do, you may catch errors earlier. This is particularly illustrative with the cron jobs. Imagine you have a visualization tool which shows you, graphically, when the cron jobs you have defined run. Immediately, you see that there is something wrong.

  Syntax highlighting, for example, is the most popular developers' visualization tool, and it indeed helps finding errors at very early stages (often within seconds of writing something wrong), before compiler or tests run.

• Continuous monitoring.

  Why the server collapsed? Wasn't there a way to see that something is going wrong before bad things happened?

  Servers should have been properly monitored. Either a person or an app should have seen that after the last change, CPU usage increased from, say, an average of 42% to an average of 100%, and could do something to prevent the server to collapse.

  This leads me to Continuous integration, and more specifically Continuous deployment. In Continuous deployment, when you make a commit, the new revision is built, tested and deployed to production on a few servers. Then, those servers are watched by an app. If the app sees an abnormal increase in resources usage or an abnormal quantity of warnings in syslog coming from those servers, the servers are reverted to their previous state.

• Strictness.

  Guidelines and rules are usually annoying (not convinced? Try to write a small C# app which conforms to StyleCop and Code analysis and has good results with Code metrics and has a decent code coverage!) but also particularly useful. This is why some projects target zero warnings with static checkers and zero warnings with the compiler set to a maximum level: errors found earlier are errors you don't have to solve when it is too late.

  While this wouldn't help with cron jobs, it helps when writing code.

Answer 5

I think the direct solution, especially to the sort of problem you exemplified, is to make mistakes conspicuous.

If you look at the two lines you gave, the difference is actually quite subtle. You have to have a lot of detailed, nuanced syntax "in your mind" when reading it if you want to catch the mistake. You also have to keep an eye out for hard-to-see visual features (* and 0 can be easy to confuse).

For example, in your example, if cron syntax was like the following:

minute=0 hour=4 day=every week=every month=every year=every

You would be less likely to confuse it with:

minute=every hour=4 day=every week=every month=every year=every

You can't really change cron's syntax, but you could write a simple CLI program for yourself that takes named arguments like above, and then pass them to the actual cron. You'd need to carefully review the code that converts from your named arguments, but you only need to do that once, so you can afford to really focus yourself and go the extra mile in error checking - print out the code, read and explain it to yourself aloud and so on.

When you have a program like this, you can also add a sanity check. After you make a new cron job, have it output:

This new job will run 1 times a day.

(perhaps it tries to find the shortest time period that results in a whole number)

Now it will be really difficult for you to make the same mistake. Your tradeoff is:

1. It now takes more keystrokes to type the commands, and they look busier (this is probably why cron was given the syntax that it was in the first place).
2. You have spent time and effort setting up the error-resistant interface model and the sanity check (but in this case you can probably do it in an hour or two).

Now, I'm not necessarily suggesting you do this for cron. That seems like it's a bit of overkill, unless you set and edit tasks on cron very often. I just used your example to illustrate the two useful tricks: Sanity checks, and preferring syntax that makes errors very conspicuous even if you don't look for them. The idea is to make sure that mistakes smell.

With syntax, the two obvious things you can do are:

1. Select tools and languages that have more error-resistant syntax, for instance, C# vs. Perl. Notice again the tradeoff between catching mistakes quickly and having to do more work.
2. Wrap poor syntax in better syntax. For instance, make your own wrapper function for a poorly named functions with ambiguous argument names, make your own wrapper program for arcane tools, use an editor that supports color highlighting and so on.

It doesn't make sense to do these all the way all the time. You want to ask yourself, what matters more in a given task: Correctness, or your time? It's no use disputing that hacking together a quick script in Perl is going to get the job faster than writing a well-structured C# program with clear OOP hierarchy. If you're not worried about mistakes, and you're in a hurry, why bother? But keep in mind that every time you take this shortcut, you are making a gamble: You wager much frustration and gnashing of teeth on the hope that you won't have to go back and extend on your project in the far future, long after you've forgotten how it was supposed to work (or that you won't realize halfway through that your original design won't work, and now must be radically altered).

Answer 6

I agree with all of the other answers, but the idea of defensive coding could also help here.

When practicing defensive coding, we write our code under the assumption that it's going to be used incorrectly; whether by us, or someone else. In this case, your shell script has been used incorrectly (run 59 times instead of once).

How could we defend against that? A simple but crude way would be to log a timestamp somewhere at the beginning of the run, and only run if the timestamp is found and more than, say, 20 hours old.

The problem with that coarse-grained approach is that we might want to manually trigger the script, eg. if some part of its functionality needs to be run again, but not all of it. In that case, a finer-grained approach would be better, eg. "if the latest DB backup is more than 20 hours old, backup the DB", "if the last announcement was sent out more than 20 hours ago, send out an announcement", etc.

Answer 7

As well as testing...

I found that printing out the code on paper, then running the code in a debugger and ticker each line once I confirm that all the values were as expected in the debugger was a great way. This gets you to look at your code in another way.

Driving the "debugger checking" with unit tests is nice...