I might be biased working in very performance-critical areas like image processing and raytracing, but I'd still say to optimize "as late as possible". No matter how performance-critical your requirements are, there's always so much more information and clarity in hindsight, after you measure, than in advance, which means even the most effective optimizations are typically applied later after gaining such knowledge.
But sometimes "as late as possible" is still pretty damn early in some peculiar cases. If we're talking offline renderers, for example, the data structures and techniques you use to achieve performance actually seep into the user-end design. This might sound disgusting but the field is so cutting-edge and so performance-critical that users accept user-end controls specific to the optimization techniques applicable to a particular raytracer (ex: irradiance caching or photon mapping), since some of them are used to waiting hours for an image to render, and others are used to dishing out enormous sums of money to rent or own a render farm with machines dedicated to rendering. There's a massive reduction in time and money for those users if a competitive offline renderer can offer a non-trivial reduction in time spent rendering. This is a sort of area where a 5% reduction in time actually excites users.
In such peculiar cases you can't just pick one rendering technique willy-nilly and hope to optimize it later, since the entire design, including the user-end design, revolves around the data structures and algorithms you use. You can't necessarily even just go with what worked well for other people since here, you, as the individual, and your particular strengths and weaknesses, factor in heavily to delivering a competitive solution. The mindset and sensibilities of the main developer behind Arnold is different from those working on VRay who used a very different approach; they can't necessarily swap places/techniques and do the best job (even though they're both industrial leaders). You have to kind of experiment and prototype and benchmark and find what you're particularly good at doing given the endless array of cutting-edge techniques out there if you hope to ship something competitive that will actually sell. So in this peculiar case, performance concerns move way up to the front as perhaps the most important concern prior to even beginning development.
Still that's not necessarily a violation of optimizing "as late as possible", it's just "as late as possible" is rather early in these extreme and peculiar cases. Figuring out when and also what doesn't need such early performance concerns, if ever at all, is probably the main challenge to the developer. What not to optimize might be one of the most valuable things to learn and keep learning in a developer's career, since you can find no shortage of naive developers who want to optimize everything (and unfortunately even some veterans who managed to somehow keep their job in spite of their counter-productivity).
As Late As Possible
Perhaps the most difficult part is to try to understand what it means. I'm still learning and I've been programming for almost three decades. But especially now in my third decade, I'm starting to realize it's not that difficult. It's not rocket science, if you focus more on design than implementation. The more your designs leave breathing room for appropriate optimizations later without changes to the design, the later you can optimize. And the more and more productivity I've gained seeking out such designs which afford me that breathing room.
Design Which Offer Breathing Room to Optimize Later
These types of designs actually aren't that hard to achieve in most cases if we can apply some "common sense". As a personal story I'm into visual arts as a hobby (I find it somewhat helps to program software for artists being somewhat one myself to understand their needs and speak their language), and I spent some time in the early 2000s using Oekaki applets online as a quick way to doodle and share my work and connect with other artists.
In particular my favorite site and applet there was riddled with performance flaws (any non-trivial brush size would slow to a crawl), but had a very nice community. To work around the performance issues I used teeny little 1 or 2-pixel brushes and just scribbled my work like so:
Meanwhile I kept giving the author of the software suggestions to improve performance, and he noticed my suggestions were of a particularly technical nature talking about memory optimizations and algorithms and so forth. So he actually asked if I was a programmer and I said yes and he invited me to work on the source code.
So I looked at the source code, ran it, profiled it, and to my horror he had designed the software around the concept of an "abstract pixel interface", like
IPixel, which ended up being root cause behind the top hotspots for everything with dynamic allocations and dispatch for every single pixel of every single image. Yet there was no practical way to optimize that without reconsidering the entire software's design because the design had trapped him into a corner where there's not much beyond the most trivial of micro-optimizations when our abstractions are working at the granular level of a single abstract pixel and everything depends on this abstract pixel. And so we gave up on the idea of optimizing the software much to handle bigger brushes and real-time filters and such and I went back to doodling with 1 or 2-pixel brushes.
I think that's a violation of "common sense" but obviously it wasn't such common sense to the developer. But it's like don't abstract things at such a granular level where even the most basic use cases are going to be instantiating by the millions, as with pixels, or particles, or tiny units in a ginormous army simulation. Favor the
IImage (you can handle all the image/pixel formats you need at that bulkier aggregate level) or
IParticle, and then you can put in the most basic and quick-to-write and simple-to-understand implementations behind such interfaces and have all the breathing room you'll ever need to optimize later without reconsidering the entire software's design.
And that's the goal as I see it these days. Excluding the peculiar cases like offline renderers above, design with enough breathing room to optimize as late as possible, with as much hindsight information as possible (including measurements), and apply any necessary optimizations as late as possible.
Of course I'm not necessarily suggesting to start off using quadratic complexity algorithms on inputs that easily reach a non-trivial size in common user-end cases. Who does that anyway? But I don't even think that is such a big deal if the implementation is easy to swap out later. That's still not a grave mistake if you don't have to reconsider any designs.