12

I'm currently getting into BDD/Cucumber and I ask myself: On which levels is it good to use?

There are the test levels (from the testing pyramid):

  • User Interface
  • Integration
  • Unit

You can apply tests on various levels and fields in the application architecture:

  • Presentation layer
    • Rules (how are business rules reflected in the UI?)
    • Workflow (how does the user get where he wants?)
    • Technical (how does the UI aid the user in getting where he wants?)
  • Service layer
    • Rules
    • Workflow (e.g. session handling)
    • Technical
  • Data access layer
    • Service gateways
    • Persistence

My current impression is:

Cucumber/BDD doesn't restrict WHAT you test with it. But best it is used for business rules and workflow. To test them you would mostly want to use unit and integration tests. UI tests can also be run with BDD, there are use cases out there. But my impression is that business rule tests are better not done on the UI, instead you can keep with just testing workflow (e.g. view transitions) and the technical level (e.g. data transfer and interactions between view and presentation logic or services) there, otherwise you would have to test business logic on both, UI and integration/unit level.

So why do I have this question here? There are contradictions in what I read so far...

  • I thought Cucumber/BDD is really just for testing FEATURES (so basically business logic?) and not for stuff like "when I click here, and i enter X, and I click OK" (workflow)?
  • I also read a lot that BDD is "of course" meant for end2end tests (my impression is that it doesn't have to)
  • On the other side I made quite bad experience with end2end tests (as a consequence I think you should just use them for a very small amount of UI test cases, basically sanity/smoke tests, see Google testing blog article about that topic)

So you can see this is about the basic questions of how a test suite is organized and architectured with BDD. What is your way to do it? It'd be a great guideline for my own implementation. What do you use BDD/Cucumber for and how? (Testing levels, architecture levels)

1
  • 2
    I think your question is interesting, however, I would heavily recommend to remove the "best practices" part, and any part which invites to a broad discussion. Some people here see this as a red flag and usually downvote/vote to close any question for "best practices" (see here why).
    – Doc Brown
    Commented Nov 20, 2017 at 7:30

4 Answers 4

14
+100

If you can answer the question, "Can you give me an example of how X should behave?" then you can do BDD.

So, if you want to do BDD at a unit level, think of some examples of how that class or unit of code behaves, then describe that example using code. You don't have to use Cucumber for that. I find it's enough to just put the Given / When / Then in comments, like this.

If the audience for the scenarios or examples is non-technical, it might be worth using a natural-language tool like Cucumber. This is typically done end-to-end through the UI, but doesn't have to be. Aslak Hellesøy, the creator of Cucumber, and various others like Konstantin Kudryashov have worked with various different flavours of stack* (seen here with Josh Chisholm), trading off confidence for speed.

You can also just write a DSL or use a DSL framework like Serenity BDD, rather than using natural-language tooling.

Testing - for instance, integration or exploratory testing - is a whole other kettle of fish. I commonly see an anti-pattern in which people have perfectly fine testing practices but get confused because it doesn't seem to fit with BDD. BDD isn't really about testing. At its heart, it's a collaborative analysis practice, which happens to produce tests as a nice by-product.

I find it more useful to think of the scenarios as "living documentation". If you're keeping your codebase clean and well-designed then scenarios or unit tests will hardly ever pick up bugs. They'll help people work out what the code should do, though, which means coders are less likely to write bugs in the first place.

*(The video is of "sub-second TDD"; BDD started as a replacement for TDD with "behaviour" a more useful word than "test", since TDD isn't really about testing either.)

Typically I'll have some examples of how the system provides value to the customer or user, written using something like Cucumber, together with class-level unit tests that provide examples of how each bit of code behaves.

For instance, I might have a couple of Cucumber scenarios showing how validation helps the user fill in the form, but the exhaustive rules will be in unit tests.

If you find that you've got too many scenarios, push some down into unit tests. Also consider whether there are areas of code which have stabilized and could be extracted out into libraries or services, with their respective scenarios running over that library/service's API, thus getting them out of your build.

And, yes, we try to avoid button clicks etc. in our scenarios. This helps to keep the code maintainable (also see the Page Object Pattern) and also provides options as to how the problem will be solved. Clicking buttons etc. describes a chosen solution, while scenarios should really be illustrating the problem.

A note for Aslak's Honeycomb pattern:

Aslak describes a pattern in which user interfaces sit outside the honeycomb (web browsers, logs, apps, etc.) with the domain layer sitting inside the inner honeycomb. Glue in the outer honeycomb allows the UIs to interact with the domain layer. He describes it:

"Honeycomb" is a software architecture pattern that makes it possible to run acceptance tests in many different modes with different tradeoffs: Speed or confidence. One layer or just some.

(Note that the honeycomb shape with its multiple sides is reflective of BDD's "outside-in", in which we recognize that there are multiple user interfaces.)

In the honeycomb pattern, we use the same tools for full-stack scenarios going through the UI (the browser, for instance) and through the domain... and, in fact, we use the same scenarios! We just choose which ones to run through the full stack, and which to run through lower layers.

Note that there are actually more layers than just the domain layer; you could for instance choose to run with an in-memory database but the full web browser; a real database but directly linked to APIs; etc..

So we'd use e.g.: Cucumber for all of them.

This is different to doing BDD at a class-level (as an equivalent to TDD).

Within the domain layer itself, there will be classes with individual responsibilities (single responsibility principle).

By thinking of how that class should behave, and providing examples of that behaviour in action, we help to drive out good design and show the value of that class, providing living documentation and keeping the code clean... which is exactly what TDD does! Except I didn't use the word "test" there.

BDD at a class level is just TDD, without the word "test".

There are lots of tools which help to do this, like RSpec and MSpec. Alternatively, I might just think of my code this way, and use a TDD framework anyway. I sometimes write the "Given, When, Then" in comments (this one's in Kotlin):

@Test
fun `should handle multiple different date formats`() {

    // Given a number of different types of date format
    val input = listOf(listOf("2017-04-03 08:25"), listOf("01 Jan 2017 01:27 PM"), listOf("8/9/2016 8:15 AM"))

    // When we parse the format of the dates
    val formatters = input.map {DateFormatParser()(it)}
    val results = input.zip(formatters, {i, f -> i.map {LocalDateTime.parse(it, f)}})


    // Then it should be able to resolve those dates.
    val expectedResults = listOf(
            LocalDateTime.of(2017,4,3,8,25),
            LocalDateTime.of(2017,1,1,13,27),
            LocalDateTime.of(2016,9,8,8,15))

    Assert.assertEquals(expectedResults, results.flatMap { it })
}

JBehave (the first BDD framework) actually started as a replacement for JUnit 3.X with this way of thinking in mind. Java's annotations didn't exist back then, so JUnit looked for methods starting with "test..." instead. JBehave did exactly the same thing, but looked for classes starting with "should...".

You can read more about the history of BDD and JBehave in Dan North's introduction and in my potted history post; you can also see there the evolution from class-level BDD, to full-stack BDD, to the natural language tools like Cucumber that we now have!

So, it's possible to do BDD at a class level too. Typically we'd use different tools. They might be the same tools; I sometimes create a nice little DSL to run my scenarios with (this one's in NUnit).

Unless it's by coincidence, though (e.g.: a class just happens to be solely responsible for some interesting behaviour seen at the UI level) the full-stack scenarios and the class-level examples (unit tests) will be different.

The main reason for this is that steps are often reused across different scenarios at a full-stack level, but when you get down to classes, they should really only have one responsibility each, with the rest mocked out.

10
  • Thanks, helps a lot for understanding! (1.) I personally favour encapsulating functionality in properly named (reusable) methods over comments. That way code explains itself (2.) "Typically end2end" you say? I'm a strong opponent of the overuse of end2end tests from experience, they tend to be flaky, hard and hard to analyse when failing (see goo.gl/2FHzr0 also), business logic should be tested below the UI I think. (3.) For me the documentation character of BDD is very important. That's why I personally would probably not push scenarios down to unit tests.
    – nepa
    Commented Dec 5, 2017 at 10:49
  • 1
    Also a strong opponent of overuse of those tests. Just saying that typically, I have a fair few of them; enough, not too many. Yes, by all means put the business logic below the UI. Think about where the documentation is most needed; if you have a very thin front end (eg: not a modern web page) then you won't need to do so much through it. As for not pushing scenarios down to unit tests... sometimes they're just examples of class behaviour anyway (see my validation example). Put it wherever it makes sense, for whatever scale you can answer the question, "Can you give me an example?"
    – Lunivore
    Commented Dec 5, 2017 at 10:54
  • @Lunivore, is there a C# Honeycomb example anywhere that you are aware of?
    – w0051977
    Commented Mar 11, 2018 at 16:08
  • @Lunivore, is it normal to test the Domain Model independently using BDD (Specflow and Cucumber)? Your answer and the hyperlink seems to imply it is. Thanks.
    – w0051977
    Commented Mar 11, 2018 at 18:17
  • @w0051977 Lots of people do run their scenarios below the UI layer; through domain model, an API, etc.. I recommend this for super-thin UIs (not JS-rich web-pages) or ones where errors matter less than data integrity (admin tools for example). It can be faster than going through the UI. I'd call it "contextually appropriate" in that situation rather than "normal" though.
    – Lunivore
    Commented Mar 13, 2018 at 12:10
1

I use BDD at as many levels as possible, using the same scenarios.

First I start with writing and validating the scenarios with stakeholders.

Then I use the scenarios to drive development of the backend (e.g. a REST API). When those tests pass, the steps code serves as great reference material for the UI developers.

So the next phase is I have the UI developers code a services level that's driven off the same BDD scenarios. Ideally the BDD tests are possible to run with a either a fake backend or the real backend. If everything went perfectly, then the UI developer doesn't need to ask the server developer any questions (the reference material I mentioned above should answer UI developers questions about the semantics of calling the API).

Lastly, the UI developers write the actual front-end which calls the services layer. Again, they use the scenarios to drive the development. The SAME scenarios used at the other two layers.

When it's all finished, the same BDD scenarios are used to test all three layers:

  1. UI
  2. UI Service Layer
  3. Backend API
1

Ask yourself, what does the BDD acronym stand for?

Behaviour Driven Development is not a testing methodology, it is development methodology that includes testing as a first class capability; specifically it uses the requirements as a script for automated testing.

It should be used for testing at the level of abstraction expressed by the requirements and should always be expressed in the language of the business requirements and used declaratively not imperativelyenter link description here.

2
  • I'd be careful with "always". For instance, this scenario is written in the domain language of a pet shop... but it's actually a toy app used to exercise a library, which solves the real requirements (automating WPF). github.com/lunivore/WiPFlash/blob/master/…
    – Lunivore
    Commented Nov 30, 2017 at 13:33
  • @Martin-Spamer, thanks for your answer! That brings things to the point but unfortunately doesn't help my intention to understand the matter better and to find a good architecture (maybe it was put a little fuzzy by me)
    – nepa
    Commented Dec 1, 2017 at 20:45
0

What levels to test on with BDD/Cucumber?

  • if possible prefer to implement automated tests via an api layer instead of gui layer because gui tests are often more expensive-to-maintain/fragile than api tests.
  • if you have to write automated gui-tests because logic is already implemented in gui-dependant code instead of a seperate gui-independant-business-api try to encapsulate the page logic into Page Object Pattern as mentioned in @Lunivore-s excelent answer to reduce maintenance-costs.
  • for unittests (test modul in isolation) i would prefer unit-tests/tdd over bdd because these are easier/cheaper to develop than bdd tests but more difficuilt to understand by a non-developer.
  • for integrationtests/usecases/atdd/gui i would prefer bdd/cucumber-style tests. It should be easy to understand these by a non-developer-business-expert but you have more software-development-overhead compared to ordinary unittests.

My prefered workflow use both bdd and unittests in two nested loops as described in Behavior-Driven Development with SpecFlow and WatiN

  • writing a failing bdd/integrationtests test

    • writing a failing unit test as part of the solution of the bdd/integration test
      • making the unittest pass
      • refactor
    • writing the next failing unit test as part of the bdd/integration test

    • unitl the bdd/integration test passes

  • writing the next failing bdd/integration tests

3
  • That's very precise, thanks! But doesn't it make sense to test on the unit level with BDD too in some cases when there are specific business rules tied to the unit?
    – nepa
    Commented Dec 5, 2017 at 10:40
  • Using bdd or unittests it is always a tradeoff between benefit/cost of development/maintanave and intended audience developer or domainexpert. Since businessrules are driven by domainexperts and probably change more often I assume that the bdd development/maintenace overhead is well spend here. May be my assumtion is wrong that bdd is more work than tdd. I have done a lot of tdd while own experience with cocumber/bdd is limited to a few examples (more read about it than doing myself)
    – k3b
    Commented Dec 5, 2017 at 12:39
  • 1
    We currently plan to set a test architecture in place where we have separated execution logic from the test workflow (so the unit test just consist of calls to service.givenLoginSucceeds(); or someUnit.whenXyz() or thenAbc() etc.). The "step definitions" which are separated by scope (e.g. business domain, system, component or unit under test) can be annotated to be usable from Cucumber or such as needed. This way we have: Self-explaining code and high reusability which is beneficial for both, conventional and BDD tests and better maintainability. That could keep the cost in balance.
    – nepa
    Commented Dec 5, 2017 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.