7

Read first: For a definition of the two terms ("profiling" and "benchmarking") and the need for distinguishing them, please read this answer to a previous question.

I have to admit that until I saw Winston Ewert's answer, I have never thought of the need to distinguish the two techniques. I simply think that "profiling" can be applied at different "scale levels" of software, and that when it is applied on the higher level, the profiling code in the lower levels ought to be turned off in order to lower the aggregate overhead.

After I reflected on the answer, it might have explained why I fell prey to micro-optimization in my previous project.

In an effort to optimize during that project, I implemented a low-overhead profiler (inserted into the source code) which is good at generating accurate profiling results at the millisecond level. I then spent all days tinkering with it, and optimized a lot of code based on the profiler's result. In the end, I was successful in reducing the computation part of the project from several seconds to less than a fraction of a second.

The next thing I learned, to my horror: when the optimized module was used in a larger project, I/O and data conversion completely dominated the module's computation time. The non-computation part is in the range of 1-2 seconds, making my optimization efforts moot.

To this date, I still haven't got a chance to do a true "benchmarking", though I am going to give it a try very soon.

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

  • I'm a bit confused. Several means more than 2, but less than many, let's assume worst case results and the computation time was 3 seconds. So the total time for your module, pre-optimization, was 3 seconds of computation + 2 seconds of I/O and data conversion = 5 seconds total. And you reduced that to 0.5 seconds of computation (again, assuming worst case results) + 2 seconds of I/O and data conversion = 2.5 seconds total. What is so terrible about a 50% reduction in total run time at scales perceptible to humans? – 8bittree Dec 4 '17 at 20:19
6

From what I've seen the "did you profile?" question always comes after "why does this run so slow?" so the "benchmarking" has been done and the result was "too slow" and now we're trying to figure out why it's running so slowly so we go and "profile" it.

Real life is usually more complicated. How fast your software is depends on the architectural decisions you make, algorithms you choose, whether or not you've correctly identified and dealt with various bottlenecks and system constraints. Getting stuck optimizing a system that isn't designed for performance is an easy trap to fall into and can suck away huge amounts of time for little reward. On the other hand, not every software has high performance as a requirement.

Profiling and optimizing before you benchmark, i.e. before you know whether or not the performance is adequate is truly falling into the premature optimization scenario.

I like this quote from Wikipedia:

“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson

| improve this answer | |
  • 2
    +1 for the Quote. It should be printed with a label printer and stuck across the top to every software developers monitor. – mattnz May 9 '11 at 1:35
1

He who does not honor the small is not worthy of the large.

Even though you may not make the specific feature significantly faster, you may safe some cycles which might help the system as a whole. Taking small bits of time off of many links in a chain may add up to being more then just combating the weakest link.

Of course you need to be careful about putting too much time in it. Small tweaks do stack up though. Small tweaks are also best done while your code is fresh. They won't typically show up in a profiler since they are small and are many all over.

First make it work then make it fast. Sometimes it's just fast enough, even though there is plenty of opportunity for improvement.

Sometimes it's not "premature" if it's hard to do later.

| improve this answer | |
  • I agree that making a lot of small optimizations on "computational" code can add up. But from my experience, I don't think that optimizations can "add up" anymore when I/O and OS effects are combined. – rwong May 8 '11 at 23:53
  • 1
    Good programmers write optimal (not optimum) code as a matter of routine. They have the scars from the resources being silently and unendingly spent all over the place. – Apalala May 9 '11 at 0:30
  • have to disagree. Until you know for certain code is a problem, leave it. What happens if you half the time a piece of code takes to run? Does your program do twice the work in the same time? What if that code only runs 10% of the time. – mattnz May 9 '11 at 1:37
  • 1
    This comes back to "is algorithm selection optimisation?" Small tweaks vs not chaining your program to a rock. To people above a certain skill level it's just good design. But to many people optimisation is going back and replacing (eg) bubble sort with quick sort. But those are the decisions where the application performance is determined. Data in memory vs filesystem overwhelms saving a few cycles getting data out of the filesystem every time. – Мסž May 9 '11 at 2:09
  • 1
    It is usually not worth sacrificing anything else for performance for single-user applications. Their performance is almost always dominated by other things, such as user input and network connections. It is pointless to micro-optimize uniformly throughout the program.. – David Thornley May 9 '11 at 14:24
1

Profiling must be against realistic scenarios (call that benchmarking if you like). (Is that a "Dugh"?)

O(n²) solutions severely beat O(n log(n)) ones by a large margin given sufficiently small datasets. That is well known.

The programs that most developers write first-hand do not scale to even one order of magnitude above. It is a project manager's responsibility to make sure that everything is tested against close-to-real-life test datasets and scenarios, and to executions/second per module when need be.

Risk management is not about being pessimist. It's about considering bad and fatal scenarios in the design and the procedures.

| improve this answer | |
  • The project manager probably only tells the developer that "You should measure this, this, and that." and leaves the details to the developers. How does the project manager find out whether the developer is doing it right or wrong, if it is up to the developer to determine whether the test scenarios and/or the test mechanism is realistic or not? (Is the average project manager usually more knowledgeable about the proper ways of performance testing than the average developer?) – rwong May 9 '11 at 1:00
  • @rwong: at some point a user sits there and goes "this is slow". Ideally the manager makes it clear to the developer what the user will be looking for. – Мסž May 9 '11 at 2:10
  • The project manager should gather the user requirements. – quant_dev May 9 '11 at 5:47
1

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

I think so, given the kinds of questions and answers that travel about these sites, and the existence of profiling myths.

It is common to hear people putting timing code into their routines, because they are unhappy or puzzled with what profilers tell them. It's also extremely common to hear people doing micro-optimization whether or not they have tried profiling.

I think part of the problem is the word "profiling" itself. It is often conflated with "measuring", when finding performance problems is not at all the same as measuring them, in my experience. Measuring can tell if what you did made a difference, but it is a very fuzzy magnifying glass for finding what to fix.

There's a very easy technique for pinpointing single-thread performance problems quickly. A small but growing fraction of programmers know it. It's based on a very simple observation. While a program is doing something it doesn't really need to do, you can see what it is by just surprising it at random. If it's wasting enough time to be worth fixing, you won't have to surprise it very many times before you spot it. Then you can see, in precise detail, what the problem is. Here's more on the subject.

| improve this answer | |
  • This works in more than just single threaded scenarios. Take a java container and execute a thread dump. If 20 of 23 threads are waiting on a lock chances are that lock is your problem. – nsfyn55 May 13 '14 at 16:38
  • @nsfyn55: You're quite right. One case where it did not point to the problem was in a manufacturing simulation where there was an asynchronous protocol between processes, and delays were caused by fast actions being prioritized behind slow ones like DB update. For that, I used a laborious logging method, which worked. – Mike Dunlavey May 13 '14 at 19:11
0

I'm actually of the very opposite mindset. The easiest way to end up micro-tuning a codebase needlessly for hours and far away from real-world user-end operations is to obsess over benchmarks.

You can end up having a performance test slow down to take twice the time and then waste hours with the team looking into and tuning it again when, in the context of the application, the small functionality being tested only takes 0.01% of the time, making trying to speed it back up again a worthless endeavor.

I actually prefer to keep a codebase's performance rather "organic", as in not trying to cement it down with endless benchmarks. At the very least, if you're going to add performance tests to your system, make sure they're high-level enough and close enough to what your users actually do with the software and frequently. I actually don't think an automated build system misses much by omitting performance tests outright as long as there are unit/integration tests for correctness, and I work in performance-critical areas.

My former company made all these performance tests for things so far removed from user end operations, like just timing how long it takes to call a function in a low-level interface a million times over, and obsessing about some performance fluctuations to those areas is counter-productive compared to actually profiling the application against a real-world use case. It even got to the point where people were wasting countless man-hours investigating slowdowns to low-level performance tests while the high-level user-end operations of the application were getting perceivably faster... still the developers were obsessed that fetching a value out of property took 30% more time than when they started even though that had very little bearing on the user. I would have much rather had them profile the application than these micro-benchmarks.

Hotspots tend to shift around to rare cases as things get more efficient in the common case execution paths, and in that former experience, so many developers wasted needless time and energy trying to optimize rare cases by obsessing over these teeny performance tests. Just as important as measuring, if not more, is thoroughly understanding how the software is commonly used. Otherwise we could be measuring/benchmarking and tuning the wrong things and, in worst case scenarios, actually making what users actually do most often less efficient in the process.

To me the real litmus test of whether you're spending your optimization time productively is if you're actually measuring real-world use cases that the users of the software commonly apply and not taking stabs in the dark (at least measuring). Typically the most counter-productive in this regard are the ones most divorced from the user-end side of things, failing to understand the appeal and common case use scenarios among the users of the software.

That said, I think "micro-optimization" is more over-used than profiling. If, by "micro", that means an optimization with no impact on the user, then even substantial algorithmic improvements should be considered "micro", but that's not how it's typically used. Often "micro" is used interchangeably with "counter-productive", when some of the most fruitful and productive optimizations don't come from algorithmic breakthroughs so much as practical algorithms that apply effective micro-optimizations (ex: improved locality of reference). I don't really care if it's algorithmic or low-level tuning of how bits and bytes are laid out in memory or multithreading or SIMD. All that should matter is if makes a real difference, and not to a minuscule function in the system infrequently called, but to the actual user of the software.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.