Well, I can't entirely agree, because you have to worry about all of it. And for that matter, one of the things I love about programming is the switches through different levels of abstraction and size that jump quickly from thinking about nanoseconds to thinking about months, and back again.
However, the higher things are more important.
If I've a flaw in a couple of lines of problems that causes incorrect behaviour, it probably isn't too hard to fix. If it's causing it to under-perform, it probably doesn't even matter.
If I've a flaw in the choice of data structure in a sub-system, that causes incorrect behaviour, it's a much bigger problem and harder to fix. If it's causing it to under-perform, it could be quite serious or if bearable, still appreciably less good than a rival approach.
If I've a flaw in the relationship between the most important data structures in an application, that causes incorrect behaviour, I've a massive re-design in front of me. If it's causing it to under-perform, it might be so bad that it would almost be better if it it was behaving wrong.
And it'll be what makes finding those lower-level problems difficult (fixing low-level bugs is normally easy, it's finding them that can be hard).
The low-level stuff is important, and its remaining importance is often seriously understated, but it does pale compared to the big stuff.