7

For my thesis, I am doing research into the characteristics of Behavior Driven Development (BDD). One of the BDD claims is that by specifying the requirements in a ubiquitous language together with the customer, the software will score better on the external quality. Another claim of BDD is that it produces clean code with a high code coverage.

In my experiment, I will perform a software project both in BDD and traditional style, so that in the end I have two end products. The second claim of BDD is easy verifiable, by checking the code against some quality metrics, like Lines of Code, Cyclomatic Complexity and Code Coverage. However, I cannot think of a good measurement for the first claim. Is there anybody out here who has done this already? Or maybe can propose an alternative experiment that can validate this claim?

Improve this question
10
  • Why not just measure external quality of products done using BDD and those done using other methods statistically? To me the artificial experiment you propose would only measure how the two methods you use perform in the specific task you have selected. You could end up with 2 results that would be both statistical outliers.
    – SoylentGray
    Mar 6, 2012 at 14:26
  • I guess "BDD" stands for "Behavior-driven developpment", in the context of this question.
    – barjak
    Mar 6, 2012 at 14:29
  • 1
    I am saying that your sample set is too small. That there already exists a dataset of projects that have been developed using various methodologies. Score those projects based on development methodology and look for your trends there.
    – SoylentGray
    Mar 6, 2012 at 16:08
  • 1
    In addition to what Chad has said about your experiments potential failings, there is no way you can compare two projects done by yourself, one will have an advantage based on lessons learned from the other that you can't account for
    – Ryathal
    Mar 6, 2012 at 16:39
  • 1
    That is true, but I don't think it is possible to analyse 1000 projects in such a detailed way. That would mean that for every project you need to check whether the requirements would meet the actual software. Furthermore, I think it's impossible to find such a large amount of open-source BDD projects, as the interest in BDD is still low compared to other methodologies. But besides that, it would be a more conclusive research, I agree.
    – Martijn van der Maas
    Mar 6, 2012 at 22:13

3 Answers 3

Reset to default
5

Have a few people act as your customers for both projects (even better if they are actual customers, i.e. would want to use your software). After demoing the project to them ask them to fill out a survey measuring satisfaction and missing features.

You may want to make the projects different, but of roughly the same size scope and complexity, otherwise your 2nd version (whether BDD or not) will be better designed and closer to the customer's wants, because you will have the hindsight of version one.

Improve this answer
2
  • Thanks for your reaction, that is indeed a good approach. I was aware of that problem you mentioned. But don't you think that it is possible to develop the software in a way that you switch everyday from development method, so that you can minimize the effect of advantage?
    – Martijn van der Maas
    Mar 6, 2012 at 14:44
  • I would spend a fixed number of hours each day on each project and alternative between them, and alternate which one you work on first for that day. Then try to code naturally and ignore any bias you have for BDD to be better (or worse).
    – Garrett Hall
    Mar 6, 2012 at 14:47
2

I wanted to tell you that all the common metrics like McCabe Complexity don't give helpful results when normalized for lines-of-code. I couldn't find the paper demonstrating that, so take it as a rumor. I did find a doctoral thesis roughly the same topic, but using TDD. Perhaps what that guy writes about code quality can help.

Improve this answer
2
  • @Bruce Could it have been from Making Software, by Greg Wilson (featured on an SO podcast)?
    – Garrett Hall
    Mar 6, 2012 at 22:15
  • @Garret: thanks for the tip, but I thought I read it on some software blog, maybe even more than one place.
    – Bruce Ediger
    Mar 7, 2012 at 2:01
2

So, based on the proposed literature I have seen that various studies use Acceptance Tests to measure the external quality of a software product. Although this is not conclusive for measuring the external quality, it does give a precise indication of the amount of satisfied requirements by the product.

The next problem in my research is the shortage of available BDD products that can be measured. That is also one of the main reasons why I've chosen to set up this experiment on a synthetic base.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.