The "VDA Automotive SPICE Guidelines" (excerpts: https://vda-qmc.de/fileadmin/redakteur/Software/Automotive_SPICE/Veroeffentlichungen/VDA_Guideline_Rules_and_Recommendations_2017.xlsx) tells to lower the score "If the verification results (…) contain only a pure passed/failed information without a supporting verification log (…)"

What is sufficient to avoid the score being lowered for this reason?

I assume that the key point of interest is the "failed" part - the log must contain information on what failed, where, and why.


testID1: PASS
testID2: FAIL at L1: var_1: expected X1, actual: A1
              at L2: var_2: expected X2, actual: A2
testID3: PASS

[edit] relevant, from the same guidelines: Test logs supplying a meaningful summary of the logged data as an adequate evidence for each test result. Sounds like "use common sense to determine what's enough".

[edit 2] I'd like a second opinion on logging PASSING checks (see discussion under Thomas' answer).
Aside from being of questionable use without the context (so, hardly a "meaningful summary"), they reduce the log's quality, with all the chaff. The test execution part alone, grows ~12 times, now being in tens of MBs. The log just doesn't feel like the right place for that.

  • 1
    I'm not sure why people are voting to close this. This is most certainly not a question about programming tools, but what information is necessary to constitute a sufficient verification log or test log under the ASPICE requirements.
    – Thomas Owens
    Sep 9 '21 at 19:22

Although I've never worked in automotive and had to deal with ASPICE, this is very similar to some standards from the aerospace, medical device, and pharmaceutical industries so it's possible to draw some parallels.

I don't think it's safe to assume that the key point of interest is the failed part, although it's probably more important for failures in order to support defect tracking and resolution (it looks like ASPICE calls this SUP.9 Problem Resolution and Management).

As far as I can tell, ASPICE references IEEE Std 829 for test documentation and refers to the Test Log definition, but IEEE Std 829 has been superseded by ISO/IEC 29119. You would have to consult the most recent ASPICE publications to see which standard is referenced and pull the appropriate standards for review.

According to IEEE 829-2008, a Test Log consists of a lot more information than your example. You would need to add information about the version of the software under test, environmental attributes, the datestamp for starting and executing tests, who executed the tests, success/failure for each test case, and the information necessary to establish traceability between test failures and an anomaly report.

To ensure that you're providing all of the information, I'd strongly recommend including the expected and actual values for all tests, including passed tests. That, plus the standard contents of the Test Log, should be sufficient to ensure that the rating for this criteria isn't lowered. However, you should also consult any internal QMS documents or templates and see if you can't provide the information that is called for by the QMS or would appear on a hand-generated paper test log template.

  • Thank you for the docs to look at. Now, re: your recommendation: I am surprised to hear about logging PASSING expectations in all tests. To me, the execution LOG doesn't feel like the right place for what is basically a poor paraphrasing of the test-spec. Why do you feel it is?
    – kaay
    Sep 10 '21 at 17:10
  • @kaay The guidance says to include more than "pure passed/failed information". I'd interpret that as requesting additional information for each test case, regardless of the execution status. From my experience with test frameworks, this is something relatively easy to get and would allow someone who is reviewing the test log to gain confidence about the coverage of testing based on inputs and expected outputs of each test.
    – Thomas Owens
    Sep 10 '21 at 19:08
  • Test coverage and sufficiency (re-reviewable at your leisure) is ensured, reviewed and approved way before they're run Param ranges, coverage or test logic for TestNameHere is easily available in convenient standardized form, rather than just expected/actual. And, the test runner tool is trusted to actually run the tests. And so on. So, we have that covered :) I feel that the log ("meaningful summary (...) adequate evidence"), in this context, only has to say "these tests have run" and it is clear that anything not explicitly reported is OK. Your thoughts, on this?
    – kaay
    Sep 10 '21 at 20:11
  • 1
    @kaay I'm not referring to reviewing test coverage and sufficiency. I'd consider it more of a convenience to include the inputs/outputs for passing tests since the tests themselves would have this information. In my experience, auditors will try to find something to make an observation about and this is very low-hanging fruit to close one potential observation. I do think the more important thing would be to make sure that the test log output includes the timestamp, environmental information, and information about the software version under test.
    – Thomas Owens
    Sep 10 '21 at 23:14
  • Wow, that feels wrong. I'm not criticising you, who might have seen this in action, only whoever thinks that bloating a readable execution summary with a worse-readable version of what is already traceable outside of the log, ACTUALLY constitutes convenience. They might use the same logic to demand a copy of the source code in the log, too. That, and even with just "passedAndThatGuaranteesAllTheConsequencesGeez / failedAndHereAreTheDetails", we're WAY past "pure passed/failed". This "low-hanging fruit" lies outside their orchard, heck, it's a different country.
    – kaay
    Sep 11 '21 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.