2

According to the standards like ISO 29119 & ASPICE, the left side of the V-Model contains Requirements, Architecture Design and Detailed Design. On the test side, there are Unit/Component Test, Integration Test, Qualification Test. And the ISO 29119 mentions tests shall be performed against WHAT is expected of the test item and this WHAT shall be described in the test basis. For Unit/Component Test, the test item is an atomic SW component in isolation and test basis is Detailed Design Document of the atomic SW component.

So, my question is: WHAT an atomic SW component shall do? --> Is this not SW component Requirements? If yes, does this mean in the detailed design documents, the atomic SW Component requirements are written down which are tested in Unit test. When the standard says perform Requirements-based tests, is this what they mean?

If we do not write atomic SW component Requirements in Detailed Design Documents, then why is there a traceability between Unit test specification and Detailed Design and not between Unit test and Requirements.

The standards are so confusing, I am lost. Please help.

1
  • 4
    Standards like ISO 29119 are written by IT bureaucrats, not by developers or testers. So it's not really surprising that they make no sense. Depending on who audits you and the tools at your disposal, one approach is to argue that the code is the detailed design and that your tooling shows the mapping between that code (design) and the unit tests and that it doesn't therefore need documenting elsewhere. You'd be right to argue it, but right and "what the rules say" don't always agree, so it can be a tough argument to have.
    – David Arno
    Jan 16, 2020 at 10:08

2 Answers 2

4

Your question is confusing!

The simple idea is that you should be able to trace a test back to a requirement. Otherwise why are you testing unrequired behaviour!

Obviously in practice, no one writes requirements as detailed as their unit tests end up being. ie

"The Addition Class will have a function Add() which shall add two ints together and return the result, which should be the sum of those ints"

They will just write

"When I add an item to the basket the total price should update accordingly"

Now I said "no-one" writes requirements at this level of detail but that was a lie. If you are launching rockets into space you probably do and you probably also want a test and you will give that test a number and check that the requirement has a test etc.

That would make you standards compliant.

Alternatively, you could also just ignore all the unit tests without requirements and only document the ones that fit the standard.

2
  • 3
    Otherwise why are you testing unrequired behaviour! When you're in more critical systems (I saw this in aerospace, I wouldn't be surprised if it's also in automotive systems), the more fundamental question is why you have unrequired behavior implemented. The only place I've seen it is in reuse of a system, but in that case, there's traceability between the reuse of the component in different systems and traceability to the design and requirements in one of those reuse instances.
    – Thomas Owens
    Jan 16, 2020 at 13:00
  • In automotive software, SW components are the lowest level software elements and specified in detailed design. We used to write blackbox software component requirements in these documents and trace the unit tests. Now, the requirements are moved up the V-Model and they plan to mark all tests as SW qualification test. and tests validating interfaces of a single SW component as integration tests and then they want developers to test how the component is implemented and mark them as unit tests. repeating 3 times with different mindsets. How can processes specify something like this? Jan 17, 2020 at 7:38
0

To get a better understanding, one needs to familiarize oneself with

  1. how the whole V-model works and what the terminology is, and
  2. how the testing according to these standards really works.

To that end, I suggest checking the ASPICE manual (version 3.1), especially check Annex D (at least in version 3.1) is quite illustrative. Also, professionally developing with ASPICE usually means, that there is someone available for the engineers to consult with (typically this should be the role of a Quality Manager).

The idea behind the V-model is top-down design, where (very crudely said) first the requirements are gathered and analysed, then architecture is created, and smallest elements of architecture are then described by detailed design. Typically all this goes without coding (as in writing actual code that will be executed, be it in compiled or interpreted way). Only the very last step of the left-hand side of the V, the detailed design is implemented in a particular programming language and the final "binary" is created (quotes are intentional as there is a historical bias towards compiled languages, but you can think of putting together all the necessary source files in an interpreted language as the "creation of binary image" if you will).

As part of this process, each software element will end up being linked to some requirements. Some requirements may be covered by multiple independent parts in the code, some pieces of code may cover multiple requirements.

Now for the testing: important is the distinction between the object being tested and the test specification (in your words the WHAT).

The object is always the binary. Depending on the level of the test in question, it may be smaller (unit tests), bigger (integration tests) or complete binary of all the software being developed (qualification tests).

The test specification is derived from the software description on each particular level your test is running:

  • unit tests verify that the binary (units) behave according to the detailed design. This thus means testing the low-level algorithms. For compiled language (assume C), the most basic test (or better even prerequisite) is "Did the source file compile into an object file?"

    Of course, that is only the first step, but this is the level of complexity unit tests work on.

  • Integration tests verify that the binary (several units integrated together) behave according to the architecture (as the name suggests, important here is the communication among the components and whether they are correctly assembled together). One thing that typically gets tested is correctness of APIs through which individual units communicate. Prerequisite: "Did all the object files link correctly into a library?"

    Architectures also often include e.g. a dynamic view (timing etc.), so there are lots of various kinds of tests.

  • Qualification tests verify whether the whole software fulfils the requirements. Only here you are actually testing the requirements, since those are for the software as a whole.

This said, I think the core of your question could be answered like this: Software units are traced to requirements, but unit and integration tests do not test requirements. It wouldn't even be reasonably possible - there rarely is a 1:1 correspondence between requirements and units. The reason behind it is, that the actual units do not depend on requirements only neither directly - they are influenced by the architecture and other constraints, which may appear because of many reasons (architecture/design decisions between several options, additional standards, hardware/system limitations).

As a side note: this whole process idea seems often menacing to new programmers, and often is frowned upon as way-too-much-overhead, but somehow it has been proven in practice. There is a reason why certain industries (aerospace, railway, automotive, healthcare) demand it. Once peoples' lives are at stake, the rules of the game are different from the "more usual" programming (commercial personal software, games, web applications).

12
  • Hello Peter, Thanks for the detailed reply. The problem is with the Terminology. ISO says a SW Unit is an atomic SW Component (lowest element in architecture). A-SPICE says SW Unit is part of SW Component where SW Component is an atomic SW Component. This changes the meaning of the whole standard from IT and other industries. So, now automotive guys start to interpret SW Unit as a C-Function or a C-File or something else. So, I feel the A-SPICE got it wrong with the terminology. They copied the text from SPICE or ISO and just manipulated terminology so this gives different meaning. Jan 27, 2021 at 13:06
  • Exactly, it is a matter of terminology - if you are in ASPICE environment, you are using ASPICE terminology no matter how much you like it (think of it as using a different language). Second, it is also a matter of what is your verification target - ASPICE is quite clear about that too. Note that what to test is given by the overall structure of the development cycle - which is an inherent part of ASPICE but (to my knowledge) not so much of ISO 29119 alone. Generally speaking, ASPICE is also a bit more detailed, which is its functional safety environment heritage.
    – peterph
    Jan 27, 2021 at 22:36
  • This terminology push in A-SPICE leads to exponential rise in workload. And unfortunately automotive V cycles are so short compared to railways, aerospace, so this change of terminology by A-SPICE without adapting text parts creates unwanted work load which is not realizable in practice. A simple example. SW Requirements shall be traced to SW Units. Now, other industries defined this so that you tag atomic SW components involved in realizing the req. Now, the same text means trace req to C-Functions or C-Files...Thats the problem when you dont adapt text and just change terminology. Jan 28, 2021 at 11:46
  • The inventor of A-SPICE V-Model writes like this: 1) Requirements shall be decomposed to the lowest level of architecture functional elements. 2) Requirements shall directly map to the lowest architectual elements. Requirements are testable at the same level as architectual elements. N.B.: To make requirements traceability easier, it is often beneficial to decompose requirements down to the same level as the software architecture and associate and organize those requirements around the architecture elements. Jan 28, 2021 at 13:45
  • Unwanted workload - please define "unwanted". Unwanted by whom? Developer/tester? Sorry, if ASPICE is part of requirements (usually it is if you are following it) you are going to follow it. If you don't like it, choose another industry. Project manager? If you are in automotive and do not understand what ASPICE means, what impact it has on a project, you are missing important part of qualification. ASPICE has its merit, the fact that sometimes people fail to see it doesn't mean the standard is wrong.
    – peterph
    Jan 28, 2021 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.