Why some consider static analysis a testing and some do not?

Question

Preparing myself also to ISTQB certification, I found they call static analysis actually as a static testing, while some engineering book distinct between static analysis and testing, which is the dynamic activity. I tent to think that static analysis is not a testing in the true sense as it does not test, it checks/verifies. But sure I would love to hear opinion of the true experts here. Thank you

Answer 1

In many respects, your question is the corollary of a number of recent discussions, about where the line is drawn.

eg

• Is verification and validation part of testing process?
• Verfication vs validation again, does testing belong to verification? If so, which?
• How to apply verification and validation on the following example

Quite simply, the line depends on which standard, and which lifecycle you are following.

Personally, as an advocate of ISO/IEC 12207, I put Static Code Analysis firmly in the Verification camp, as it is a review activity, whereas Dynamic Analysis is a testing activity, therefore a Validation phase.

As an aside, I had never heard SCA referred to as Static Testing, so I Googled... Wikipedia defines it, pretty much as a synonym of SCA. I prefer to use the terms Static Analysis, and restrict testing to cover the actual execution.

I'm not that familiar with ISTQB so I can't draw the specific line for that certification for you.

Answer 2

One of the traditional definitions of software testing is execution of the program with the intent of finding bugs. This was Glen Myers definition, for example. We pretty much followed that definition in Testing Computer Software. I still see this--Wikipedia, for example, says "Test techniques include, but are not limited to, the process of executing a program or application with the intent of finding software bugs (errors or other defects)."

Under that definition, static analysis cannot be testing because it analyzes the code without executing it.

The definition that I currently use is that testing is an empirical investigation of a software product or service in order to learn quality-related information about it. (See for example, the BBST courses (my free course videos for BBST are at http://www.testingeducation.org).

Under this definition, static analysis might be testing. If you think that SCA is an empirical activity (in other words, something like running an experiment), then you should think that SCA would be testing under my definition. In contrast, if you think that SCA is more like theoretical analysis than like running experiments, then you should think that SCA is not testing.

I don't personally have an opinion. I use SCA while I am writing code and it feels to me that it is part of what I do when programming. It doesn't much feel like testing. But I also do a lot of unit testing and monitor code coverage of the unit tests and when I do it, that also feels like part of an integrated programming activity, not like testing either. Mainly, I don't care whether people call SCA (or unit testing) testing or not. The more important question to me is whether they do it, and if they do, whether they are any good at it.

There are several other widely used definitions of software testing. Some people (and some organizations) insist that they have the One True Definition. And I suppose it is the One True One for them. But there are enough different One True Definitions that I find it more useful to accept the diversity as a fact of life and work with people by asking them what they mean when they say things.

So, depending on your favorite definition of testing, SCA is or is not testing (or is ambiguous), but under someone else's favorite definition, the answer might be different. If you're studying for the ISTQB exam, the relevant definition is the ISTQB definition. I don't remember it, but you can look it up in your study materials.

By the way, static analysis is a broad term. A program that analyzes code to check for certain specific coding errors (or even nonconformities with a specified coding style) is doing static analysis. So is a person or program that analyzes code to determine whether an algorithm is theoretically correct (and whether the implementation conforms logically to the algorithm). This latter case is called a proof of correctness. This is an instance of static analysis, not the entirety of it.

By the way, proofs of correctness cannot prove that the program as run will be correct. They can prove that certain types of errors cannot be present. However, if you analyze the source code (e.g. code written in Java), you have not analyzed what actually gets run, because the Java gets translated into other code by a compiler. Additionally, you don't know how this Java code will interact with the memory manager, the printer driver, the video display, etc. Those interactions might cause a program failure but the static analysis will be blind to them. Another class of increasingly common risk involves multithreaded execution. What looks to you like one linear sequence of commands can be executed partially in parallel. Sometimes, things get done in an unexpected order of execution (the wrong thing gets done first, so perhaps a memory location has outdated information or information that shouldn't have been put there yet) The result is a failure, even if it wouldn't have been a failure on a different computer. Finally, I have never seen an error-free specification, so the idea that a proof that some code matches the specification perfectly doesn't assure me that the program is error-free.

So.... formal analysis is useful, but it cannot replace all of (the other types of) testing.

Answer 3

I agree, I do not view static analysis as testing. Static analysis is more powerful than testing because it is a formal proof that a certain piece of code matches its specifications. Therefore, it guarantees that a piece of code does what it should do, for all inputs. In other words, it can guarantee the absence of errors.

Testing, on the other hand, is done on a limited number of different inputs. Therefore, it can only indicate the presence of errors, not their absence. This is the biggest difference between static analysis and testing.

However, static analysis is very difficult, time-consuming and fundamentally impossible to perform for all programs due to the incompleteness of first-order logic.