0

I was reading up about branch predictions and cache misses, and have decided to read what every programmer should know about memory. But I'm not sure how much low level knowledge do I need to have to write performant code.

One classic example is accessing the memory in the wrong way, accessing column by column when it's row-major would cause a cache miss on every lookup.

I ran a benchmark of 10 000 by 10 000 matrix, and the correct order runs in 97ms while the wrong order runs in 1535ms.

While it's evident that accessing the memory in the right order is much faster, but by how much? In this example, it's only faster by 1.44s. So a program written by a programmer without such memory knowledge is probably only penalized by up to a few seconds.

I have 2 questions.

  • How much low level knowledge does a competent programmer need to have?
  • How much time saved through optimization is considered worthwhile? At least a minute shaved? Half a minute? Or any amount of time saved?

Some people think that it's totally not worth it to optimize just to save a few seconds or even milliseconds. But what do you all think?

Improve this question
13
  • 7
    Does you writing the optimized code cost your employer more money than the speed increase saves? That's really the only valid measure of what is worthwhile or not. True, this estimate is often hard to make, but it's the only correct one. Every hard-and-fast rule that involves quantities such as "half a minute" cannot possibly be correct.
    – Kilian Foth
    Nov 23, 2015 at 11:00
  • 1
    Additionally to Kilian, it also depends on what type of a system you are developing. Lately I have been working with backend developers, whose task is to make backend service for multiple partners. And this service has to be fast. Therefore saving miliseconds is all we care about. If your aim is to make the application as fast as possible, saving time is everything, but if you are more focused on the result and just need to get it done, you are likely to expect some drawbacks.
    – Andy
    Nov 23, 2015 at 11:06
  • It depends. Really can't answer this.
    – Pieter B
    Nov 23, 2015 at 11:07
  • 2
    Possible duplicate of Is it always wrong to optimize before profiling?
    – gnat
    Nov 23, 2015 at 11:32
  • 2
    @LeeYikJiun the problem is that you ask a very general question. what is a: "competent programmer"? I know guys that make millions while programming but will give you a glazing stare when you start about cache misses. Also, when it's worth it, I mean there is no general answer to that. In a windows environment, a buttonpress that takes 1 second is waaaaaaaay too long while when I'm building a report people don't care about 10 or 20 seconds more. I've even had instances where I had to "slow down" the process because my customers didn't believe the computer could be that fast.
    – Pieter B
    Nov 23, 2015 at 13:22

1 Answer 1

Reset to default
2

Its not only a question of how long a piece of string is, but what that string is used for.

Sure, spending a month to optimise a second off your code execution time is probably worthless. You could have found some slower code that was easier to optimise, or add more features. But sometimes you know something is slow and you need to modify it to make it faster and you have to experiment a bit to achieve this - for example a colleague years ago spent several months implementing a caching mechanism to our product (that was far too slow at the time). It added a 10% increase to overall speed. Then we discovered the MS_DTS setup we were using was incredibly inefficient, changed it to DCOM and achieved a 100% speed increase. What was certain was that our product was far too slow and we all needed to spend a lot of time making it faster in every way.

The other aspect is how much time is saved cumulatively - shaving a fraction of a second off a query might sound useless, but if that query is used repeatedly and often by every user, those fractions are going to add up to a fairly worthwhile server performance improvement.

So its difficult to answer your question without knowing your specific circumstances. You just need to trade off the performance impact against the cost of development.It can be useful to give a "cost to the user" to make this trade off easier to calculate - saving a second on how long it takes the start menu to show might not sound much, but if you have a billion users it might be worth the effort.

Improve this answer
1
  • Thank you for your answer. I don't really have any specific circumstance now and I understand that my question is quite vague and general, but I just want to gather opinions. I think a good way to think about it is not improvement per instance but the cumulative improvement.
    – Lee Yik Jiun
    Nov 23, 2015 at 12:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.