2

During unit testing it is possible to estimate the code coverage to see which share of the code base is covered by the tests.

For one part (the simple calculatable one) of a risk estimation we need to estimate the impact when a single line or a variable changes on the code base.

E.g. if a variable changes and a pointer to that variable is passed around or used in several places that would result in a rather "impactful" change.

The approach I am looking for is some mixture of introspection and code coverage, but I could not find anything useful so far.

Sure, small changes can completely break the code while large changes might not have an important impact. But this is only supposed to be one part of the impact analysis, the other one is done by a reviewer.

If such a software or approach does not exist, how is this handled in rather large projects? Is is useful at all or would you recommend a different approach?

1 Answer 1

2

First of all, doing this algorithmically likely runs into undecidability issues. Suppose we have a program made of some statements: { S1, S2, S3 ...}. Suppose S1 is changed, but S2 doesn't halt. In that case, S3 is not impacted at all. As part of proving that S3 is affected by a change in S1, we have to show that S1 and S2 halt, so that control can pass into S3. Or else that their halting status is altered; e.g. S3 wasn't initially reached, but the change to S1 made it reachable, or vice versa.

Qualitatively, the way we can evaluate the impact of a change to a program is to consider what parts of the program's control flow graph are being modified, and how that affects the data flows.

Under what conditions, what inputs, does control reach the modified nodes of the program, and how do they affect subsequent control and data flows. When we are working with code as software engineers, we don't work with formalisms of this sort. We have tools for inspecting code in various ways, like "what are all the callers to this function".

We can take shortcuts in the analysis that err on the side of caution. For instance, we can assume that a change in that function affects that entire function as such: everything which calls it with any inputs is affected. Strictly speaking, that may be far from true, because within the function, that small change has a particular effect on the control graph and data flows, and that effect does not necessary affect all ways of using the function.

So, assuming the worst: all ways of using the function are affected, we can move on to a global assay of the impact: what calls it and under what conditions. Here we can also take "bounding box approximation" shortcuts: like if a certain module or functional area of the program calls the modified function, that entire area is impacted, and in all ways of that functionality being activated.

When it looks like the impact of a change is significant, then we can go back and fine-tune the assumptions: which ways of activating the functional areas or modules, with what inputs, actually "step on" the change by calling the modified function?

Speaking of "step on", empirical approaches are possible. You can place an abort instruction into the altered piece of code, and make a build of the software. Then see under what conditions it aborts. For instance, which test cases in the test suite blow up. Instead of aborting, you can log a message along with a detailed call stack trace, so then you have information about the kinds of ways in which that code is reached at run-time.

The abort or log should be made conditional, if possible. The code should test "am I being invoked in such a way that I will produce the new behavior?" in that case, log. Otherwise proceed silently.

Run the software with this logging in place for a while and gather the information; then you have some idea about the impact.

In some situations, it can be useful to make an intermediate change first which doesn't actually change the behavior of the program except for only producing a trace whenever a situation occurs under which the proposed change would alter the behavior: "am I being invoked in such a way that if the proposed change were implemented, there would be a difference?"

This doesn't give you the full picture; you still need the program analysis to get an idea of the ripple effect of that change. But at least you know that for those cases of the software which don't produce any traces, there is no impact.

Ripple effects could be studied with toolchain-specific analysis tools. If I wanted to know what are the places in a C program which will all be affected if a certain datum is changed, I might build it on a platform where the Valgrind debugger is supported. A that point in the code where the change is made, I would use the Valgrind API to mark the altered memory as uninitialized. Then Valgrind will report a diagnostic whenever any sort of conditional execution takes place based on the value of that datum (thinking it is uninitialized). In any specific test case I execute, I would see the exact impact of the change: the places in the code that are branching differently based on the altered datum, or passing that datum into an OS system call.

Here is a "live demo" of this. My test program is this:

#include <stdio.h>
#include <valgrind/memcheck.h>

int var = 42;

void change_var(void)
{
  var = 73;
  VALGRIND_MAKE_MEM_UNDEFINED(&var, sizeof var);
}

void print_var(void)
{
  printf("value of var is %d\n", var);
}

void switch_on_var(void)
{
  switch (var) {
  case 42: puts("forty-two"); break;
  case 73: puts("seventy-three"); break;
  }
}

int main(void)
{
  print_var();
  switch_on_var();
  change_var();
  print_var();
  switch_on_var();
  return 0;
}

I build it on Ubuntu 18 like this:

$ gcc -g var.c -o val
$ ./var
value of var is 42
forty-two
value of var is 73
seventy-three

Now let's run it under Valgrind:

$ valgrind --track-origins=yes ./var
==6062== Memcheck, a memory error detector
==6062== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6062== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==6062== Command: ./var
==6062== 
value of var is 42
forty-two

So far so good! The initial, unchanged value of the variable is produced without any diagnostics. Once it is changed, the same functions produce errors:

==6062== Use of uninitialised value of size 4
==6062==    at 0x489BE0B: _itoa_word (_itoa.c:179)
==6062==    by 0x48A0725: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x489BE13: _itoa_word (_itoa.c:179)
==6062==    by 0x48A0725: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x489F83A: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x489F8FB: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x489F91D: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x489F94F: vfprintf (vfprintf.c:1642)
==6062==    by 0x48A7455: printf (printf.c:33)
==6062==    by 0x10875F: print_var (var.c:14)
==6062==    by 0x1087EB: main (var.c:30)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
value of var is 73
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x108783: switch_on_var (var.c:19)
==6062==    by 0x1087F0: main (var.c:31)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
==6062== Conditional jump or move depends on uninitialised value(s)
==6062==    at 0x108788: switch_on_var (var.c:19)
==6062==    by 0x1087F0: main (var.c:31)
==6062==  Uninitialised value was created by a client request
==6062==    at 0x10871D: change_var (var.c:9)
==6062==    by 0x1087E6: main (var.c:29)
==6062== 
seventy-three
==6062== 
==6062== HEAP SUMMARY:
==6062==     in use at exit: 0 bytes in 0 blocks
==6062==   total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==6062== 
==6062== All heap blocks were freed -- no leaks are possible
==6062== 
==6062== For counts of detected and suppressed errors, rerun with: -v
==6062== ERROR SUMMARY: 10 errors from 8 contexts (suppressed: 0 from 0)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.