1

Virtually any book dealing with software testing mentions that user acceptance testing (UAT) is an ultimate validation activity, often quoting Boehms informal definition: "Validation: Are building the right product?"

But how come that Boehm himself considers UAT a verification?

V Model Diagram

GUIDELINES FOR VERIFYING AND VALIDATING SOFTWARE REQUIREMENTS AND DESIGN SPECIFICATIONS. B.Boehm:

From the V-chart, it is clear that the key artifact idstinguishes verification activities from validation activities is the software requirements baseline. This refers to the requirements specification which is developed and validated during the Plans and Requirements Phase, accepted by the customer and developer at the Plans and Requirements Review as the basis for the software development contract, and formally change-controlled thereafter.

By definition,verification involves the comparison between the requirements baseline and the successive refinements descending from it — the product design, detailed design, code, data base, and documentation — in order to keep these refinements consistent with the requirements baseline. Thus, verification activities begins in the Product Design Phase and conclude with the Acceptance Test. They do not lead changes in the requirements baseline; only to changes in refinements descending from it.

On the other hand, validation identifies problems which must be resolved by a change of the requirements specification. Thus, there are validation activities which occur throughout the software life-cycle, including the development phase. For example, a simulation of the product design may establish not only that the design cannot meet the baseline performance requirements (verification), but also that the performance requirements are too stringent for any cost-effective product designs, and therefore need to be changed (validations).

The only explanation I can think of would be that Boehm did not consider Acceptance test as a USER acceptance test but as some kind of final check against the specification.

In addition, I believe this original article every book refers to is much clearer than those artificial definitions of V&V (that are, in addition, wrong from this point of view). According to Boehm, when I check the products against the requirements, it is a verification. When I test the product for its use, I validate it as I might find something that leads to changes in the requirements.

Of course Boehm's original idea contradicts some standards, such as ISO 12207, in which validation is to confirm that the requirements for a specific intended use of the software work product are fulfilled. Which means checking against the spec.

Most books says that verification is only static and the code is not run. I believe that according to Boehm, it is not true. E.g.:

Software Engineering And Quality Assurance, A.A.Puntambekar: The verification activities fall into the category of static testing.

Effective Methods for Software Testing: Includes Complete Guidelines,William E. Perr: Verification testing - testing in a static mode - ..... Validation testing (e.g. unit, integration ...)

I would conclude that despite many authors' opinions, I believe VERIFICATION includes testing which is done in order to check the correspondence of the specification and the program. Validation includes testing as well but it would not be based on the specifications.

  • The author describes his reasoning clearly and completely in the passages that you quoted. – Robert Harvey Dec 11 '14 at 14:59
  • @Robert Harvey Exactly. So why every software testing book considers UAT a validation activity? Why most of these books say that verification is only static while validation dynamic testing. Boehm does not imply anything like that. – user144171 Dec 11 '14 at 15:17
  • By static, do you mean in accordance with the existing baseline requirements, and dynamic in accordance with changes to the baseline requirements? – Robert Harvey Dec 11 '14 at 15:20
  • 1
    I see your edit, but you still need to clarify what you mean by "static." – Robert Harvey Dec 11 '14 at 16:31
  • 1
    Please join us in The Whiteboard, where a discussion of your question is taking place. – Robert Harvey Dec 11 '14 at 16:56
5

In the opening sections of the paper that you quote, Boehm states:

There is very little agreement in the software field on the definitions of the terms "verification" and "validation."

Boehm defines the terms verification and validation as he intends to use them in the rest of the paper, in order to be able to clearly present the rest of the writing without ambiguity:

Verification - to establish the truth of the correspondence between a software product and its specification. (Note: This definition is derived from the Latin word for "truth," veritas. Note also that data bases and documentation are fit subjects for verification, as well as programs.)

Validation - to establish the fitness or worth of a software product for its operational mission. (Note: This definition is derived from the Latin word for "to be worthy," valere.)

I think that the Guide to the Software Engineering Body of Knowledge can provide some insight into this as well:

Classical treatments suggest that “verification” deals with static evaluation methods and that “testing” deals with dynamic evaluation methods. Recent treatments, including ISO/IEC draft 29119, are blurring this distinction, though, so testing standards are mentioned here.

What all of this says is that, at the time Boehm wrote his paper, there wasn't a widely accepted singular definition for what is "verification" and what is "validation". Even today, that's still true.


Now that we know that there is some ambiguity in the terms "verification" and "validation" (and how "testing" fits into these), we can look at the diagram that Boehm presented.

Something that Boehm didn't discuss, but I've seen in multiple sources (including ISO/IEC 12207), validation specifically requires that the validation activities occur in the operational environment and can achieve its intended purposes, from the user's perspective. This may be implied by Boehm's definitions, but it's not clearly stated.

Boehm also used the term "acceptance testing". You are interpreting this as "user acceptance testing". There are other kinds of acceptance testing, such as factory acceptance testing (FAT). In my experience, a FAT is performed by the developing organization at the developing organization's site, with the procedures reviewed and accepted by the customer. This is a check to ensure the complete system (all software and hardware) are fully functioning, meet the specification, and a final inspection prior to delivery to the customer's facility. Based on Boehm's definitions, this can be a verification activity the developing organization can not fully exercise the product within the operational context.

What you are considering "user acceptance testing" is likely "OT&E" - Operational Test and Evaluation. This is indeed a validation activity since the customer and user are able to see the product within the operational context and say if it does or does not meet their needs.

  • Thanks, it does make sense. However, when doing e.g. exploratory testing, what would that be? It seems that whether the findings results into the specification change determines whether I did validation or verification. – user144171 Dec 12 '14 at 7:14
  • @user144171: What is the aim of the exploratory test you are doing? What is your mindset when doing it? If the aim/mindset is to find deviations from the requirements, then it is verification. If the aim/mindset is to see if the product is usable, then it is validation. Or would "OT&E" also become verification if an implementation bug was found? – Bart van Ingen Schenau Dec 12 '14 at 10:15
  • @Bart van Ingen Schenau That is true. I spent 2 hours on Google books today and I am so mad that so many authors put "verification" into "static analysis only" while it clearly needs to be dynamic testing too. Probably the older definitions are flawed. I believe that when I test the UI against the spec, I do the verification, not validation as most sources suggest. – user144171 Dec 12 '14 at 10:29
1

It might be useful to examine the testing artifacts that ISO 12207 demands under Verification and Validation:

Verification

7.2.4.3.2.3 Code verification.

The code shall be verified considering the criteria listed below:

a) The code is traceable to design and requirements, testable, correct, and compliant with requirements and coding standards.

b) The code implements proper event sequence, consistent interfaces, correct data and control flow, completeness, appropriate allocation timing and sizing budgets, and error definition, isolation, and recovery.

This describes Unit Testing and Static Analysis, among other things. Unit Testing and static analysis are below the red requirements baseline; they are verification activities.

7.2.4.3.2.4 Integration verification.

The integration shall be verified considering the criteria listed below:

a) The software components and units of each software item have been completely and correctly integrated into the software item.

b) The hardware items, software items, and manual operations of the system have been completely and correctly integrated into the system.

That's Integration Testing, still part of the Verification process.

Validation

7.2.5.3.2.3 [Testing Includes]:

a) Testing with stress, boundary, and singular inputs;

b) Testing the software product for its ability to isolate and minimize the effect of errors; that is, graceful degradation upon failure, request for operator assistance upon stress, boundary, and singular conditions;

c) Testing that representative users can successfully achieve their intended tasks using the software product.

Testing of this kind is seldom embodied in the original requirements specification. It normally arises as a result of user testing and feedback, among other things. It rightly belongs above the Requirements Baseline, and is part of the Validation process.

ISO 12207 does not appear to make a distinction between "static" and "dynamic" testing.

As to Acceptance Testing, my view is that if a requirement is a valid requirement, it is testable. Acceptance testing is, therefore, the process of verifying the requirements via tests, and belongs below the Requirements Baseline as part of the Verification process.

Accordingly, changes to the software requirements that arise as a result of user studies belong above the line, as part of the Validation process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.