Can software performance monitoring/profiling be automated to a high degree, as in unit testing?

Question

Unit testing consists of executable code which exercise a certain functionality and then assert some conditions. The results are either Pass or Fail.

• The objective is always to have all tests passed.

Software performance profiling usually requires help from some automated test/profiling tool, but the results seem to always require human interpretation, because:

• There are no clear pass/fail objectives.
• There are many facets (dimensions) in software performance, and one needs to consider the overall performance profile (the full picture; no pun intended) in order to interpret the result and make a decision.
• There are many pitfalls in software profiling, such as results skewed by profiling overhead, unintended interactions between modules or operating system, test cases not representative of needs of customers.

Now that unit testing has been heralded as a major breakthrough in promoting the credibility of software engineering, is it possible to find the next silver bullet for the software performance issues?

Clarification on "automated to a high degree":

• What I mean is that unit tests that were written long time ago (and updated as the software is developed) will remain useful indefinitely. During the better part of the software's production lifetime, there is little to no need of modifying the unit tests unless there are big changes to functionality.
• On the other hand, performance testing seems to require adjustments and re-running continuously, even well into the production stage.

Related:

• How is performance testing integrated into your product lifecycle?

Answer 1

Firstly, you should not use profiling to determine performance. Profiling is intended to be used in order to identify the parts of your program which take the most time. When measuring performance, we only care about the speed of the whole program taken together. Profiling in that case only skews results. Instead, performance should be tested by benchmarks. The program processes some task, and we just measure how long it takes.

Secondly, you don't pass or fail performance tests. What we want to do is identify when something has made the program run slower. By running a benchmark against different revisions, you can easily obtain performance data for each revision. From there is would be fairly simple to automatically raise a warning on significant performance degradations.

http://speed.pypy.org, has done something like this. You can see the performance of PyPy as charts which show its performance. You can look at each benchmark and how it has changed over various revisions. You can actually see some revision which markedly decreased performance but were later fixed.

Answer 2

@Winston has some good points, but I must disagree in one context: when the specification (feature) includes service-level agreements (SLAs) then you should absolutely automate performance profiling (aka stress testing) if at all possible, and the software must pass by meeting or exceeding the metric specified by the SLA.

Answer 3

Why worry about Unit performance at all? Whats the problme it solves? Unless you have a historical problem with a particular unit, in which case perfformance just becomes another functional requirement. I suggest that spending one more micro second thinking about performance is too much - you will spend all your time micro-optimising code that hardly ever runs, and then wonder why you are shipping late, over buget, incomplete and buggy code.

Better slow than never I say.......