Writing Acceptance test cases

Question

We are integrating a testing process in our SCRUM process. My new role is to write acceptance tests of our web applications in order to automate them later. I have read a lot about how tests cases should be written, but none gave me practical advices to write test cases for complex web applications, and instead they threw conflicting principles that I found hard to apply:

• Test cases should be short: Take the example of a CMS. Short test cases are easy to maintain and to identify the inputs and outputs. But what if I want to test a long series of operations (eg. adding a document, sending a notification to another user, the other user replies, the document changes state, the user gets a notice). It rather seems to me that test cases should represent complete scenarios. But I can see how this will produce overtly complex test documents.

• Tests should identify inputs and outputs:: What if I have a long form with many interacting fields, with different behaviors. Do I write one test for everything, or one for each?

• Test cases should be independent: But how can I apply that if testing the upload operation requires that the connect operation is successful? And how does it apply to writing test cases? Should I write a test for each operation, but each test declares its dependencies, or should I rewrite the whole scenario for each test?

• Test cases should be lightly-documented: This principles is specific to Agile projects. So do you have any advice on how to implement this principle?

Although I thought that writing acceptance test cases was going to be simple, I found myself overwhelmed by every decision I had to make (FYI: I am a developer and not a professional tester). So my main question is: What steps or advices do you have in order to write maintainable acceptance test cases for complex applications. Thank you.

Edit: To clarify my question: I am aware that Acceptance testing should start from the requirement and regard the whole application as a black box. My question relates to the practical steps to write the testing document, identify the test cases, deal with dependencies between tests...for complex web applications

Answer 1

In my acceptance suites I have stayed away from using technology specific controls i.e for web applications don't use css dont use html elements if you need to fill in a form do the specifics in the steps to setup the SUT not the actual acceptance tests

I use cucumber for my acceptance and have the following

Given A xxx 
And I am on the xxx page
And a clear email queue
And I should see "Total Payable xxxx"
And I supply my credit card details
When I the payment has been processed
Then my purchase should be complete
And I should receive an email
When I open the email with subject "xxx"
Then I should see the email delivered from "xx"
And there should be an attachment of type "application/pdf"
And attachment 1 should be named "xxxx"
And I should be on the xxx page
And I should see my receipt

this example is back by a web application but I can still use the test to test against a desktop application as the steps are used to setup the SUT not the acceptance tests

this test sits at the end of a purchase which goes

Generate -> Confirm -> Payment -> Print Receipt

the test above is for the payment step the other steps are setup in other tests due to the application being able to setup into these states with data or http actions in this case the payment has a given which does the confirm steps and the confirm does the generate steps so they are a bit brittle at the minute

Answer 2

First you need to define Acceptance Testing.

What you seem to describe is integration or system testing.

So while I do not 100% agree with the definitions on wikipedia, they are still largely valid.

Basically the purpose of acceptance testing is to verify that the 'business' processes that make use of the piece of software you built actually work as intended and are fit for purpose - with real life data. So as such you do not build test cases like you do unit tests or the rest. It is not supposed to be engineered quite in the same fashion.

The question to ask is "how is the system used?". So lets test it the way it is supposed to be used. Of course now you put your engineering hat back on and go through the business requirements religiously to derive your test cases. That assumes that you have well written business requirements.

If you don't, it is not too late, you must sit down with the user(s) or their representatives (and the business analyst and the technical design person) and write down what they expect the software to deliver in business terms (with the obvious caveat that this is too little too late, but it is better to start late than never - and of course don't introduce new features). This is what your test cases are going to be.

Another way to go about it (again if you have such a document) is to go through the user manual. Although this is one step removed from actual business requirements so only to be used if all else fails.

When you go buy a car, you do not generally go very deep under the hood (unless you are a car mechanic that is). You just sit at the wheel and check for comfort, usability, looks, sounds... i.e. the general stuff. You generally trust that if the car got to be in your hand in the first place (at least for a new car), it is generally safe and well built (there is a warranty, you have done your home work and looked at the specs...). So now you check whether this is the car that you will want to drive for the next few years.

Same with software.

Answer 3

The conflicting information can be frustrating, and difficult to generalize and apply to your specific situation. Ergo, you may have to do what works best in your context.

I'm not a big fan of long test documents, and have used visuals quite effectively for a few smaller projects. Try that? Like a flow chart (or any other UML diagram like a state diagram, etc) instead of using only text? Indicate inputs, outputs, conditions, loops, lanes, states, interactions with other components etc, and then indicate if they are successful, failed, transferred, other(?) based on your criteria.

Might be a bit of work at the beginning, but may help keep you sane in the long run. Whatever method you choose, the more you work with it, the better you will get at it.

HTH, and good luck!

KM

Answer 4

I think you've nailed down some good criteria already. Your second point is a good way to define the scopes for your tests, and I suggest also testing for error conditions and bufixes (I advocate that every bug fix come with at least one new unit test). It may seem overwhelming now, but just dive in, and after getting a bit of experience, recognizing what makes for good tests will become easier.