Earlier, I asked a question about how to measure the quality of a project. The outcome of that question was that the quality of the project can be divided into two parts:

  • Internal quality (code quality, measurable by code quality metrics)
  • External quality (Acceptance test, how well the software meets the requirements)

So based on that, I want to set up some research and validate the outcome of the project. The problem is, I will conduct this research on my own, so it's not possible to run the project once in BDD style and the other one in waterfall by myself. It's also not possible to compare BDD and waterfall projects on a larger scale, due to the fact that there are not enough BDD projects that can be measured because of the age of BDD.

So, my question is: did anybody face this problem? How could I execute my experiment in such a way that it is of scientific value?

  • Sounds like you're trying to prove something and probably redo the last 20 years of development and change from waterfall to BDD. Commented Mar 11, 2012 at 23:37
  • 1
    Do you need empirically sound data? Why do you need to collect this data yourself, rather than leveraging the body of existing research on BDD and how it compares to other development methodologies?
    – Thomas Owens
    Commented Mar 11, 2012 at 23:41
  • 1
    You may want to add an important quality measure not listed above, namely, real-life results evaluation (includes user satisfaction, fault measures, failure measures, response to change, etc.)
    – NoChance
    Commented Mar 11, 2012 at 23:43
  • This research is meant for my thesis, and for writing my paper it needs to have empirical data. So it's also not really possible to just use the existing research of BDD. There is not really empirical research on BDD as well.. Commented Mar 11, 2012 at 23:49
  • 7
    I'm not an academic, but from an engineering process perspective, a single project isn't enough to be valuable as research. When I review research into process improvement, it's often on the scale of multiple projects over a period of years. I'm not confident enough to present this as an answer, since conducting research isn't my speciality. However, it any work product produced under the conditions you describe wouldn't be valid or useful to me, as a process engineer, attempting to propose a process improvement, based on what you describe in your question. That might be something to consider.
    – Thomas Owens
    Commented Mar 12, 2012 at 0:09

6 Answers 6


It's impossible to generate large enough data sets in a research lab, let alone all by yourself, for validating development processes. Your options are:

  1. Partnering with industry. For starters, you'll need to coach them on BDD at the onset of the project and they must be willing to adopt it -- a tall order! You'll also need a few years to run your experiments on several projects and possible in several companies, possibly spanning different industries! For the control Waterfall projects, you could just rely on existing data.

  2. Using student projects. This option is practical, but it's not as reliable as the scale of projects have to be small. Hey, you gotta do what you gotta do! Perhaps if you have access to graduate students -- it might better.

If you must continue, option 2 seems to be the only practical option. However, you might want to reconsider your research topic!

Good luck!


You have set yourself a very difficult task!

Each software project is basically unique, a different set of people implementing a different set of requirements for a different customer in a different environment.

Plus commercial organizations will not make any internal data on project costs, effort or even success or failure public. In fact nearly all projects for businesses are successful purely because "success" is redefined as whatever was delivered.

Government organizations on the other hand must operate in public and figures on costs, resources and success/failure are published. At least at the beginning of projects they tend to be quite open about what they are doing and how, so between press releases, audit reports and published accounts (and Freedom Of Information requests at a last resort) you should be able to build up a pretty good body of data to analyze.

The caveat is government projects (at least the ones my taxes pay for) have a long and sordid history of failure and generally follow unique and uniquely incoherent project management styles.


Waterfall is an ALM, BDD-1/BDD-2 is a method for ongoing acceptance testing they way TDD is a method for ongoing unit testing.

The two cover different aspects and are not mutually exclusive.

E.g. You could any of these ALMs {Waterfall, UP, SCRUM} with or without BDD.


The real value of BDD is in discovering the tricky situations where scenarios aren't obvious - the unknown unknowns of a project.

Since every project involves doing something new (otherwise you'd be able to just buy it off the shelf or use open-source) there are usually a few of these discoveries. They're the thing which take up the most time on projects. ("If a project has no risks, don't do it" - Waltzing with Bears).

I therefore despair of ever comparing one project to another, especially if you use BDD to help you make those discoveries, because they'll be radically different for each project - and if you try to redo a project, you'll have found all the discoveries already and be biased accordingly. (Those discoveries are why Waterfall doesn't work so well, by the way.)

Instead, I'd ask the main stakeholders how happy they were with the internal and external quality of the software they got from a bunch of BDD and Waterfall projects on a scale of 1 to 10, and compare the results. That's about as objective as you're likely to be able to get, and should work across multiple projects.


I'm purely going on what I read here. Why don't you compare BDD to TDD? Take a group of first year CS majors and compare their ability to use unit testing with and without BDD. Do they develop a better understanding of the process?

There is merit in trying to get some data on agile to waterfall methods, but that's beyond a graduate thesis IMHO.


I think rrufai's 2nd option might be the best one to pursue. However, I would change it up a bit.

Find some mid to senior level programming classes and split the students into two groups - waterfall and BDD. Depending upon the size of the class, you could easily have multiple teams for each approach.

Next, take specific challenges from Google's CodeJam project. Assign the teams so that you have at least one waterfall and one BDD team per challenge. Add additional requirements such as documentation, progress indicators, tracing / logging, etc... to increase the number of features that have to be designed and built into the project.

The benefit of using the Jams is that you have a built in validation for their code. You'll have to determine the evaluation criteria for the additional requirements.

Ideally, you'll have each team address a few of the challenges in their assigned style. Then you can switch design methodologies and have them address a few more challenges.

  • Running multiple challenges gives you the benefit of additional "projects" to build up the comparison, but it pulls down the overall size of the project. Depending upon the number of classes that you can borrow like this, it may make sense to run some semester-long projects and some short iterators like the Jams.
    – user53019
    Commented May 8, 2012 at 15:18
  • I don't see the value in running tests on toy projects. With projects of that small of size; the programmer's skill level will have far more impact than any process that is followed.
    – Dunk
    Commented May 8, 2012 at 17:24
  • @Dunk - I struggled with that a bit too. The Jams are intentionally small projects. There is a trade-off between having a project of a reasonable size that it can be completed vs. something of greater complexity but may not be accomplished in the limited time for that audience. Using the Jams provided a known means to verify the "primary requirement" of the task was complete. Additional work would be required to validate the add-on requirements and to make sure they increased the scope enough to expand beyond a toy categorization.
    – user53019
    Commented May 9, 2012 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.