If you're working in genuinely performance-critical areas, then you can't put efficiency off as an afterthought. It's one of the most critical things to think about when designing early on in those cases and in ways that relate to the maintainability of the end result.
You can't design and implement a large-scale server and just start off writing easy, well-documented code that just uses blocking functions for everything with a global thread lock that locks the entire system to process each individual client request while not putting any thought whatsoever into shared state, thread contention, and asynchronicity. Such is a recipe for disaster and a need to redesign and rewrite the bulk of the nicely-documented code you wrote in ways that could lead to the most difficult-to-maintain codebase imaginable, plagued by race conditions and deadlocks as a result of trying to achieve the required efficiency in hindsight, as opposed to having just having thought about efficient, simple, working designs upfront.
A game development team 8 months into production with an engine that only goes 2 frames per second on their beefiest hardware with 32 cores while having a tendency to stall for 15 seconds every time the screen gets busy is unlikely to instantly get a usable product by just fixing one little localized hotspot. Chances are that their design is FUBAR in ways that warrant an epic revisiting of the drawing board and design changes which could cascade into every corner of the codebase.
With John Carmack, he talked once about how a tech demo has to run at the minimum of hundreds to thousands of frames per second in order to integrate it into production. That's not an unhealthy obsession with efficiency. He knows upfront that games need to run, in their entirety, at 30+ FPS for the customers to find it acceptable. As a result one little aspect like a soft shadow system can't be running at 30 FPS, or else the game as a whole can't possibly be fast enough to provide the required realtime feedback. It's unusable until it achieves the required efficiency. In such performance-critical areas where there's a fundamental requirement for efficiency, a solution that fails to achieve adequate speed is actually no better than one that doesn't work at all, since both are completely unusable. And you can't design an efficient soft shadow system that runs at hundreds to thousands of frames per second as required for a realtime game engine unless you put a predominant amount of thought upfront as to its efficiency. In fact, in such cases, 90+% of the work is oriented around efficiency since it's trivial to come up with a soft shadow system that works just fine at 2 hours per frame using path tracing, but you can't expect to tune it to running at hundreds of frames per second without a totally different change in approach.
When efficiency is a fundamental part of an application's design, you can't expect to achieve efficiency in hindsight without losing dramatically more time than you saved by ignoring it, since you can't expect to achieve a working design in hindsight. No one says ,"it's okay to put off thinking about design till later. Just document your code well and you can come up with a proper design later." But in performance-critical architectures, that's what you are effectively doing if you don't put a great deal of care and thought into efficient designs upfront.
Now that doesn't mean you have to micro-tune your implementations right off the bat. For implementation details, there's a lot of room to iterate towards faster solutions after measuring provided that the design won't need to change, and often that's the most productive way to go about it. But at the design level, it does mean you have to put the sufficient thought into how the design and architecture will relate to efficiency right from the start.
The key difference here is design. It's not easy to make big changes to designs in hindsight as designs accumulate dependencies, and the dependencies will break if the design changes. And if a design has a requirement to be reasonably efficient or, in some cases, that its quality is largely measured by its efficiency, then you shouldn't expect to be able to achieve a proper design as an afterthought. With any competitive products where efficiency is a huge aspect of quality whether it's operating systems or compilers or video processors or raytracers or game engines or physics engines, thoughts about efficiency and data representations were meticulously thought about from the very beginning. And in those cases it's not premature optimization to put so much thought into efficiency upfront. It was placing such thought exactly at the most productive time to do it, and right from the beginning.