How quickly, and how severely, should one sound the alarm when things look suspicious?

Question

I presently have the task of baby-sitting a business critical batch ETL system, querying the database of this system every morning for inconsistencies, and rerunning jobs that have failed.

I am not the original designer of the system or of the database, but have made several changes to both, and am reasonably comfortable with the codebase and the data model.

The system has a reputation for being unreliable. Improvements have been made, but we are still a long way from hitting the targets we need to hit. This has led to trust issues for both the system and the team involved. Hence, the system now has a baby-sitter who makes sure it behaved well every morning while we work on hardening it up.

With this in mind, I have come up against a dilemma that my team is split on. Whenever the system shows inconsistencies or failed runs, how rapidly and how severely should we sound the alarm?

Here are the factors that have gone into the discussion thus far:

• If we do not know what is causing a symptom, but the symptom looks severe, do we hold off on sounding the alarm, or sound the alarm with a lesser severity until we know the cause and discover it really is that severe?
• Say we know the cause of something causing a symptom, and the symptom is severe. We have a strategy to mitigate that symptom, but the solution has a possibility of one or more failures before a success, and will nevertheless take time to implement. Do we hold off on sounding the severe alarm until we complete the mitigation steps, and still see the data going sideways?
• Say we know the cause of something causing a symptom, but the cause is out of our control, and we will not be able to mitigate the symptom until the cause is rectified by a third-party. Does this deserve the same severity of alarm as if the data loss were the responsibility of our code.
• If we sound false alarms too often, we run the risk of losing even more trust, because now we seem incompetent about the failure modes of our system. Is it worth delaying alarms, or tempering their severity, to avoid losing trust in our ability to know when and how our system is breaking? Is following up on high severity alarms to lessen the severity level enough mitigation, or has the damage already been done when the severe alarm goes out?

The reason I ask this is because I sounded what turned out to be a false alarm this morning, due to some data that looked initially to be way out of what I expected. Further research demonstrated that it was just a run against unusual data, and so the inconsistency I saw was to be expected. Along with this, I had to go back and rerun a few things that failed, and had to wait for these to finish before my data consistency checks would mean anything at all.

Another member of my team called me on this, and made the point that I should not have sounded the alarm, and should've instead either said nothing until I was certain, or reported a less severe status until I was certain that the data was in a severely compromised state and that it was our responsibility for it being that way. By doing so, I am not helping us restore trust.

So, with this in mind, I'd like to ask the following questions: when would you sound the alarms, and at what severity, for batch job issues like the one I describe above, knowing that it has a history of problems? Also, are we focussing on the wrong things in building back trust by trying to assure we send off alarms as little as possible?

Answer 1

If you want to build trust I suggest that you not ask us, instead ask the people that consume your alerts.

Now is a great time to bring it up. Briefly explain the situation and ask if they think the alert should have been sent out as it was or not. Asking for and acting on user feedback is a great way to get the correct answer for your situation and build the trust you need.

Answer 2

When you sound an alarm, you should first, to the maximum extent possible and practical, know:

1. What the problem is,
2. How to fix it, and
3. How long it will take to fix it.

The severity of the problem will dictate, to a certain extent, how much due diligence you apply to these three reporting areas. If the problem is really severe, and you already know that, then all of your resources should be focused on solving the problem, not reporting in detail about it.

But don't hide things; this will undermine your credibility more than having an "open-door policy" about problems. Just make sure you understand each problem, so that you you know what you are talking about when you do report it. This will help avoid the "false alarm" problem you described.

Ultimately, your team should be trusted to fix problems, not spend their time telling management what the problems are. You gain that trust by providing full disclosure and being diligent about solving problems. Over time, you will develop that trust, and management will let you know how much notification they want.

Answer 3

First and foremost, I would encourage you to remind yourself and those that you work with that the trust in the product and trust in your team are mutually exclusive. Your first task is build trust in your team in order to build the equity you need to influence the product in a way that you can finally address the shortcomings that cause people not to trust it.

To build trust in your team, what I recommend is a focus on three key aspects to your operations:

• Communication/Transparency
• Consistency
• Reliability

I have no idea what your current operations is like, but if it is not already backed by an issue tracking system, and change management system, install one. You can install anything from RT, Bugzilla or FogBugz, or use a hosted service like Lighthouse. Either way, you need a system like that to provide visibility into your process, to provide adequate tracking for specific events, and to act as a knowledge base for how to resolve issues of a specific type.

Next, you need a "Run Book." A Run Book will document in a more coherent fashion, the steps to take when an issue is detected. A Run Book provides step-by-step instructions to people on your team for identifying, diagnosing, fixing, and/or if necessary escalating the issue to someone else on the team who can bring more experience to bear on the problem.

Also, consider publishing a system status page which clearly and succinctly states whether the system is ok, in distress, in critical condition, etc. Give other people in the organization the tools they need to communicate to their constituencies, be it the CEO or customers. The ability for someone to go somewhere, ascertain if things are ok or not, can be life saving to those responsible for responding to customer issues and/or questions. "Thank you for letting us know. We are aware of the problem and are actively working to resolve it."

Finally, conduct and publish postmortems for each new type of event. Distribute the postmortems to select individuals in the company in timely fashion.

This may seem intimidating, especially if it is a lot of extra work/process than what you are currently accustomed to. Just remember, these systems all go to support the three values your teams to invest in building.

1. Creating documentation and a system to track issues when they occur gives everyone in the organization access to knowing what is going wrong, who is working on it, and what is being done to fix it. The postmortems allow your team to document causality and hopefully illustrate over time that your team is not the root cause of the problem, and that your team is actively engaged in the process of making sure bad things don't happen again.

2. The documentation and run book help your organization to execute consistently. Some problems are fixed quickly and some can take days. But consistency in how problems are approached and resolved can help get people off your back when things go awry because they know from experience (even passively) that your team executes well time and time again.

3. Finally, these systems help demonstrate your reliability by telegraphing your work into the rest of the organization. It shows leadership through crisis and the ability to keep a cool head.

All of this goes towards building trust in your team. Then, once people trust you, you will hopefully have the clout necessary to influence the product team to make changes so that you are not constantly engaged in fire drills.

Answer 4

Mission Critical Systems need Mission Critical Monitoring

It sounds to me like the systems that you are monitoring are mission critical and will cause the business both financial and legal stress if they go down.

Generally It is better to have a false alarm than to have no alarm and open yourself up to litigation or to be operating under the false pretense that your system is running but is actually offline, but no-one has noticed yet. I.e. A payment gateway system is currently offline, yet transactions are failing without throwing visible errors (The errors are in a log file somewhere).

Automated Monitoring

You should have an automated alert system such as Nagios which is monitoring all mission critical parts of the system. Having someone review logs for unexpected data is ok for the short term, however being able to have an immediate alert if a payment gateway vendor or a mission critical VPN goes offline can mean a big difference.

Nagios is not just for monitoring if systems are online and pinging. You can write plugins from your software to report status back to Nagios. I have had Nagios monitor servers running MSMQ and report message queue status

Trust through Transparency

Once you have automated monitoring in place then there is a semblance of trust in the system and inherently the team and the application. If something goes offline it'll be alerted immediately to and it can be handled.

Risk management is a business decision not a technical one

The issues that software monitoring can't help with is corner case logic errors which can and will happen in large complex systems. Generally if a bug of this severity continues this can cause legal issues. In which case the business is better off shutting down that portion of their system until it can be addressed.

However this is a business decision and in every instance the business should always know whats going on with the system so they can perform risk management.

Answer 5

This has led to trust issues for both the system and the team involved.

By doing so, I am not helping us restore trust

when would you sound the alarms, ...

Trust == Transparency. Visibility. Clarity.

Nothing to do with alarms. Sounding an "alarm" is a mistake except in the rare event of personal injury, data security breaches or financial control failures.

Take the threat level down a few notches. Facts are facts. Try to stick to the facts.

Take "alarm" out of your vocabulary. Switch from "alarm" to "fact" and simply report the facts without any anxiety or stress or alarms.

Answer 6

For me it's simple:

If you are going to miss you SLA, you warn.

If you are experiencing loss of data integrity, you warn loudly, with option to take offline. Often, yesterday's valid report is better than today's wrong report.

If you are experiencing reduced connectivity, you warn.

If you are going to make the data system unavailable for maintenance, you warn.