What Is The Process of Creating A Bug Bar?

Question

In my company, we're working on adopting the Microsoft Secure Development Lifecycle, and part of the MSDL process involves establishing security and privacy bug bars at the onset of a project. I've heard the concept of a bug bar prior to MSDL, and I understand that it's essentially a definition of what level/amount of bugs you're willing to accept in a final product, but I've never understood how to creating a bug bar for a project. Are there any well documented processes for establishing bug bars that I can learn from?

I've tried doing some Googling for examples or real world scenarios of projects defining bug bars at the onset of a project, but I can't seem to get any good results or tips on the process of establishing a bug bar. There are MSDL examples of what appear to be completed bug bars, but I'm interested in learning about the process of defining something like this. For example, for those that have done something like this before, have you defined bugs bars in a very specific way (e.g. saying "There shall be no unauthorized file system access: reading from the file system"), or have you taken a deferred approach of saying when we find bugs we'll rate them on a 1 to 5 scale (5 being the most severe), establish now that no bugs above a 3 shall be shipped, and leave your ranking of bugs until they're discovered? I feel like the trying to do the former is a foolish and impossible activity, but the later is prone to bias towards leniency when a project is coming down to the wire.

Can anyone provide me with a well documented approach for defining/creating bug bars?

Answer 1

Let's start with a basic definition for Bug Bar:

Quality gates and bug bars are used to establish minimum acceptable levels of security and privacy quality. A project team must negotiate quality gates (for example, all compiler warnings must be triaged and fixed prior to code check-in) for each development phase. A bug bar is a quality gate which is used to define the severity thresholds of security vulnerabilities—for example, no known vulnerabilities in the application with a “critical” or “important” rating at time of release. The bug bar, once set, should never be relaxed.

When a software user files a bug report, they must assign a STRIDE category to the bug, whether the bug is a client or server bug, and what scope the bug affects. Users can be software developers and QA staff. STRIDE stands for:

• Spoofing
• Tampering
• Repudiation
• Information Disclosure
• Denial of Service (DoS)
• Elevation of Privilege (EoP)

The relevant "scope" values are

• Client - Spoofed trusted UI in common/default scenario
• Client - Spoofed trusted UI in specific other scenario
• Client - Spoofed UI as part of a larger attack scenario
• Server - Spoofed specific user or computer over secure protocol
• Server - Spoofed random user or computer over secure protocol
• Client - Tampered trusted data that persists after restart
• Client - Tampered data that does not persist after restart

From there, a matrix is created that assigns a level of severity to each combination. The possible values for level of severity are:

1. Critical
2. Important
3. Moderate
4. Low
5. None

Example entry in matrix (there can be several such entries):

STRIDE Category: Spoofing
Scope: Client - Spoofed trusted UI in common/default scenario

Description: Ability for attacker to present a UI that is different from but visually identical to the UI that users must rely on to make valid trust decisions in a default/common scenario. A trust decision is defined as any time the user takes an action believing some information is being presented by a particular entity—either the system or some specific local or remote source.

Severity Level: Important

Microsoft claims that they fix all bugs that are higher than "low" importance before deploying.

Further Reading
Add a Security Bug Bar to Microsoft Team Foundation Server 2010
New SDLC: Security Development Life Cycle