This has happened to most of us...

You come to work one day. Everything seems normal - the sun is shining, birds are chirping, but you notice a couple of weird things on your way to work that remind you of the déjà vu cat in the Matrix.

You get into the office and there are a lot of phones ringing - but it could just be that they are doing a new sales promotion. You settle in, when you notice a dark cloud hovering over you.

It takes you a couple of moments, but you recognize the cloud is your boss. Usually he checks on you every morning with his "Soooo Peeeeter, how about those TCP/IP reports?" routine, but today he forgot everything about common manners and rudely invaded your personal space. No "Good Morning", just some drooling, grunts and curses. He reminds you a bit of a neanderthal who is trying to get away from a cyber -toothed tiger, fear and panic all compressed in a tight ball. You try to decipher the new language that he created since yesterday and you start understanding that something bad happened overnight - the production system went down.

Now, your system is usually used by clients during regular working hours from 9-5, but for whatever reason you didn't get any alerts on your beeper (for people under 30 - a beeper was like a mobile phone that could only ring and tell you who beeped you). You'll need to remember to charge it next time.

So it is now 8:45am, and the system MUST be up at 9am. Every 10 seconds, your boss lets out yet another curse which communicates to you that another customer is having problems getting into the system. Also, several account managers are now hovering over your boss trying to make him understand how clients are REALLY REALLY suffering.

Everyone is depending on you to get the system up ASAP and at the same time is hindering your progress by constantly distracting you.

How do you keep cool in a situation like this?

  • 35
    Step One: Compose a 300 word post on programmers.stackexchange.
    – kubi
    Feb 25, 2011 at 11:02
  • 8
    Not saying it is happening right now. Wait let me check...
    – Mag20
    Feb 25, 2011 at 11:03
  • 1
    Is this a problem unique to developers? If something you are responsible for isn't working you need to be able to deal with the pressure regardless of what that "thing" is.
    – ChrisF
    Feb 25, 2011 at 12:32
  • 1
    I've found that in my own experience, very few software houses, large and small perform any disaster recovery drills. I'd put this to your boss. If things you do a drill, then you know what to expect and you know can get a feel for the response times. You can also evaluate if any of the processes can be automated. What happens if you lose power? What happens if a fire starts in the office, do you have an offsite location? Are your servers hosted in house or externally etc. Really, you need to stress for a continguency plan to be put in place. Feb 25, 2011 at 14:46
  • 3
    This reads like the start of an entry on TheDailyWTF! Feb 25, 2011 at 17:14

11 Answers 11


In the situation, ask your boss to help you by keeping all the other folks away from you (which gives him something to do somewhere else).

When you get it up and running again, ask your boss for a meeting to evaluate and establish procedures for avoiding this happening again.

  • 1
    +1. Disaster recovery drills are good ways to gauge reactions and reponse times. Shame I don't see enough of it done. Feb 25, 2011 at 14:47
  • @DP yah, but we can't do that because it would mean the people and equipment aren't available for actual emergencies while the drill is going on (yes, I've heard that argument more than once). Of course if there were enough people, and equipment, you could train one team on one set while the other is on duty...
    – jwenting
    Apr 26, 2011 at 7:47
  • @jwenting sounds like saving on the fire alarm.
    – user1249
    Mar 15, 2012 at 10:42

The first thing to do is to remove the distractions as politely as possible. Nobody can work with someone wittering in your ear about how bad it is for your customers. This is of course easier said than done if your boss is a maniac, but if that is the case, you might want to consider finding another job anyway.

Then make a quick appraisal of the real loss the error is causing and how (if at all) it can be mitigated quickly. With a bit of practice, you can also do a quick check on log files, which you'll need to form a plan of action.

If the problem is complex, concentrate on the most severe part of it. Think two or three steps ahead before you leap into action. Also, make sure you know how to back out of any plan before you act.

And the most important thing: Don't panic!


Situations like this are common in industrial control systems. The production line goes down in the middle of the night, the company is typically losing hundreds, or even thousands, of dollars per minute, and they're looking at you to fix the problem. You handle it thus:

  1. Explain to them what you know
  2. Explain what you don't know (but need to know to solve the problem)
  3. Explain how you're going to find out what you don't know
  4. Give them an estimate of how long that's going to take (use a range)
  5. Ignore everything around you while you focus on following through on your plan

First thing is to have repeatedly practiced disaster recovery (without people standing over your shoulder) so you know exactly what steps you need to take to diagnose and fix the problem without having to resort to questions on SO to find out what to do. Once you feel confident in your recovery skills, the pressure and stress is much much lower.

Next is to get the people out of your hair while you work. Your boss wants something he can go to his boss with. Give them some information on what you intend to do and how long it may take to do and then regular progress reports especially if you find something that means it will take considerably longer than you told them. Yes progress reports take time awy from fixing it, but hovering bosses and users take even more time away. Me, I go for the progress reports each time. Once they are confident you will keep them up-to-date, they will trust you to do your job more and leave you alone more.

If users are going to be blocked for some time, then send an email to them if this is an option or put up a notice on the website, saying the site is down for maintenance and when they should be able to try back. (This is possibly one task you can give your boss to find someone to do to keep him out of your hair as well.) People are less cranky about not being able to log in when they know someone is working on the problem. When things are fixed, if you sent out an email, email the same group to tell them it is fixed. Can't tell you how many times I've seen people forget this and users still think they can't log in when they can. The goal isn't just to get things up but to get people working with the system again.

Breathe deeply (deep breaths are calming) and plunge into the problem. It's good to have the things you need to do written down somewhere because in an emergency sometimes your brain synaspes don't pull up information as quickly as normal. You don't want to look like an idiot muttering: "I know we have a log, where the hell is it?"

If you are in a job where you support production systems, it is best to be the kind of person who reacts well in an emergency in general. I'm not sure you can learn this really. If someone riding a horse in front of you fell off (a not-so-random example taken from my life) and was lying bleeding on the ground, are you the kind of person who stands there with his mouth hung open or are the one who calls the ambulance, puts the pressure bandage on the bleeding and directs someone to catch the horse? If you are the first type of person, perhaps this isn't the right line of work for you.


Tell them this is a good reason why you need a backup server, and by that I mean a second server that runs the same as the primary one that can be switched to immediately if the first one goes down.

  • I've seen a backup server switched on, and it had the same problem as the primary server. It doubled the hardware cost, added to the configuration cost, and was a total waste of expense. If you're doing high-availability work, then sure, but you have to right-size your hardware to the problem. Feb 25, 2011 at 12:56
  • one (extreme) example of the backup system being affected by the same bug as the primary system is Ariane 5 Flight 501 Feb 25, 2011 at 16:46

It's bad enough when you're surrounded from all sides by people who are angry with you for a problem you created, though it's twice as bad when it's a problem you didn't create. It's happened to me more than once that the client just configured it badly, meaning the fault is in communicating with the client (whether the fault is the client for not listening or the marketer for not explaining well, you'll never know).

How do you explain that they screwed up? Never an easy task, especially when your boss is breathing down your neck because he doesn't know any better than to assume the customer is always right.

So how do you keep cool in a situation like this? Politely remind your boss that the sooner you get to work, the sooner this problem will get fixed.


By seeing this event as an opportunity to show how valuable I am (to the business) by getting the production system back running as quickly as possible (if not before 9am ;-)).

Obviously, hoping I didn't break it in the first place ;-)

  • sh_t happens
  • there must be a solution to a problem
  • if someone in the world knows the solution, I can be one of them
  • if there is no solution, panic does not help
  • again, sh_t happens

Well definitely ask your boss that you will get back to him when you get the issue resolved; though in these sort of situations the management usually gets other people involved to get it resolved ASAP and then dea, with the "concerned" person later... That is the norm with any firm regardless of the industry; as for business the Customer is usually the King!!


Situations like that just motivate me more to have a thorough documentation of everything, and a thorough plan for dealing with any kind of situation.

Even if we can not predict every possible problem, but we can work our behind's off, in being more prepared, and organized, and documented.

  • 1
    I've never solved a production problem (i.e. system down) by using documentation.
    – Marcie
    Feb 25, 2011 at 17:14
  • 1
    No, but if you need to look something up, like specs, table definitions, server settings, it pays to have it documented.
    – crosenblum
    Feb 25, 2011 at 17:20

I spent 8 years doing maintenance on B52G bombers on a 5 minute alert for World War Three. That puts everything in perspective for me.

A down production system is important, but it is not going to kill millions or billions of people.

Find out what is wrong, find the cause, fix it. Establish clear communications with those that matter and keep them informed. Tell your boss what you are doing and when you will be able to update him can prevent a continuing steam of "is it fixed yet" messages and conversations.

do a through post-mortem and figure out how to prevent and limit the effects of such incidents in the future.

If you are on call, having a dead battery on a cellphone or beeper is extremely unprofessional. This is made up general scenario, but if this happened to a person working for me, there would be a serious discussion and if it was repeated, they would no longer be working for me. Yes I am a hardass.

Not the answer you're looking for? Browse other questions tagged or ask your own question.