Acceptance tests done first...how can this be accomplished?

Question

The basic gist of most Agile methods is that a feature is not "done" until it's been developed, tested, and in many cases released. This is supposed to happen in quick turnaround chunks of time such as "Sprints" in the Scrum process.

A common part of Agile is also TDD, which states that tests are done first.

My team works on a GUI program that does a lot of specific drawing and such. In order to provide tests, the testing team needs to be able to work with something that at least attempts to perform the things they are trying to test. We've found no way around this problem.

I can very much see where they are coming from because if I was trying to write software that targeted some basically mysterious interface I'd have a very hard time. Although we have behavior fairly well specified, the exact process of interacting with various UI elements when it comes to automation seems to be too unique to a feature to allow testers to write automated scripts to drive something that does not exist. Even if we could, a lot of things end up turning up later as having been missing from the specification.

One thing we considered doing was having the testers write test "scripts" that are more like a set of steps that must be performed, as described from a use-case perspective, so that they can be "automated" by a human being. This can then be performed by the developer(s) writing the feature and/or verified by someone else. When the testers later get an opportunity they automate the "script" for regression purposes mainly. This didn't end up catching on in the team though.

The testing part of the team is actually falling behind us by quite a margin. This is one reason why the apparently extra time of developing a "script" for a human being to perform just did not happen....they're under a crunch to keep up with us developers. If we waited for them, we'd get nothing done. It's not their fault really, they're a bottle neck but they're doing what they should be and working as fast as possible. The process itself seems to be set up against them.

Very often we end up having to go back a month or more in what we've done to fix bugs that the testers have finally gotten to checking. It's an ugly truth that I'd like to do something about.

So what do other teams do to solve this fail cascade? How can we get testers ahead of us and how can we make it so that there's actually time for them to write tests for the features we do in a sprint without making us sit and twiddle our thumbs in the meantime? As it's currently going, in order to get a feature "done", using agile definitions, would be to have developers work for 1 week, then testers work the second week, and developers hopefully being able to fix all the bugs they come up with in the last couple days. That's just not going to happen, even if I agreed it was a reasonable solution. I need better ideas...

Answer 1

first, get rid of the split between 'testers' and 'developers'. Everyone tests

second, in TDD, the developers code the tests before they code the feature/story

What you have described is not TDD [it may be Scrum though; Scrum is a project-management methodology independent of the development methodology; Scrum is not relevant to your problem]

Scenarios where automated testing is impossible are exceedingly rare. Scenarios where automated testing is difficult, expensive, or inconvenient are much more common - but it is precisely these scenarios where automated testing is needed the most.

From the vague description, I assume the software is drawing something on the screen. If what is being drawn is determined by data, formulas, or functional steps, then at least write automated tests that test to the level of the data/formulas/functional steps. If the screen output is deterministic (the same steps result in the same drawing output every time) then test manually once, take a screenshot, and let future tests compare the output to the verified screenshot. If the screen output is nondeterministic and not governed by data, formulas, or functional steps, then you're in that rare area where automated testing may be impossible. But I doubt it.

I'm guessing that the only reason testing has not been automated so far is that the developers don't care about it. In TDD, the developers do the testing, so they feel the pain of the boring repetition of testing the same 62-step process a hundred times until all the bugs are gone. Nothing will get an automated testing solution developed faster than making the developers do the testing.

Answer 2

My teams found that TDD wasn't adequate for designing GUI tests. All of the automated GUI level testing frameworks we used required code to be written before the test. So we switched to Behavior Driven Development. Sure, the tests aren't automated, but it did give us a way to measure if the UI was "done-done" from the beginning.

Answer 3

We find ATDD for the GUI to difficult/expensive.

We usually do the front-end first. Since we write webapps it's usually in HTML/CSS/JS, and then we get sign-off on the the look, flow, etc. These will eventually get translatedinto templates with the appropriate parts replaced with dynamic bits.

Once the UI mockup is complete we write tests that mimic a web request. XPath is used to assert the data exists in the correct HTML tags.

We find this style of testing gives us good value for the time we spend. We still assert the data is returned, and some general structure about the html. Since we already worked out the look beforehand through the mockup stage we don't worry about trying to assert pixel placement. We just look at the page as we develop both as part of development as well as a double check.

For GUI non-web dev I would probably try add some hooks to make the front end scriptable. I would probably also mockup the UI first as well, even if on paper.

Answer 4

One thing we considered doing was having the testers write test "scripts" that are more like a set of steps that must be performed, as described from a use-case perspective, so that they can be "automated" by a human being. This can then be performed by the developer(s) writing the feature and/or verified by someone else. When the testers later get an opportunity they automate the "script" for regression purposes mainly. This didn't end up catching on in the team though.

This is more or less what we do. Every feature is developed in parallell with the test case (script) and no feature is 'done' until all three steps is done: development, test case and testing. All test cases are written from the requirements by testers and checked with the developers, so we all have a common understanding of the problem. Like your 'every other week' suggestion, except that when the developers are working on the feature the testers are working on test cases and when the testers are testing the developers are investigating the next feature.

I think the biggest problem you have is that

[t]he testing part of the team is actually falling behind us by quite a margin. [...] If we waited for them, we'd get nothing done

Since the test team is so far behind I really think you ought to wait for them. I'm sure there are something productive you could do - such as develop some features that are easy to test but takes a lot of time to develop. It can also help to take test time into consideration when planing a sprint, neither developers or testers should be overworked or idle.

Answer 5

 > Acceptance tests done first…how can this be accomplished?

I prefer a bdd-tdd-style of development as described in this article: Behavior-Driven Development with SpecFlow and WatiN.

The Example uses .net for development with SpecFlow + NUnit + WaitN. If you are developing with (j)ruby/java you can try cucumber + junit + waitr

Answer 6

Acceptance test-driven development (distinct from, but complementary to test-driven development) is a great way to ensure that you have the requirements nailed before you begin coding.

On my team we (product owner, QA analysts and developers) sit together and use FitNesse two write acceptance tests. These sessions are usually driven by QA but we all participate. These won't run straight away since some development effort is needed to add the fixtures (the glue between the wiki pages and the production code). These tests form the acceptance criteria for a story. FitNesse is designed to interact with the layer below the UI.

As developers we then work to make these acceptance tests pass. We use TDD for this, but TDD should not be confused with ATDD.

Once the FitNesse pages turn green we are almost done we still have some work to do (exploratory testing, usually lead by QA, and so forth).

Answer 7

Maybe writing coded UI tests can help you out. If you are working with Visual studio 2010 you can use an add-in called "coded ui test" for recording and coding a very specific script for interacting with the UI of your app. Recording an interaction into a script introduces an extra level of abstraction which guards you aganist small changes in the UI. However introducing this form of testing also introduces the burden of maintaining the tests.

Answer 8

8 years later the situation is pretty clear, everyone who tried to do it failed.

Upfront and maybe all kind of testing doesn't work efficient for the topmost interaction GUI layer. It's a bit better for WebUIs because they are much simpler in general but for GUIs when mouse movement and Drag&Drops are involved: don't waste to much time with testing.

Test the lower level of the logic and states you use in GUI. Here some GUI scripting products and processes can help you to get something done. But most time instead of testing use DesignByContract or also known as use tons of assert statements and invariant checks. And leave them in code while beta testing. Make sure you have enough asserts that any error can be found and fixed immediately.

For acceptance tests, just write them down as you write down normal specifications in traditional waterfall systems. Just rewrite them in every cycle.

Agile,TDD and BDD is only a useable development method for a subset of software. Dont try to use it outside the domain where it was intended to be used.