I always found the term, "micro-optimization", rather ambiguous. If some instruction-level changes to memory layout and access patterns makes something 80 times faster from a disciplined professional measuring their hotspots with no reduction to algorithmic complexity, is that a "micro-optimization"? To me that's a "mega-optimization" to make something 80 times faster on a real-world use case. People tend to talk about these things like such optimizations have microscopic effects.
I'm not working in gamedev anymore but I work in VFX in areas like path tracing, and I've seen many implementations of BVHs and K-D trees out there which process ~0.5 million rays per second on a complex scene (and this is with multithreaded evaluation). Roughly speaking I tend to see a straightforward implementation of a BVH in a raytracing context at under 1 million rays/sec even with multithreaded evaluation. Except Embree has a BVH that can process over 100 million rays on the same scene with same hardware.
That's entirely due to "micro-optimizations" that Embree is 200 times faster (same algorithm and data structure), but of course the reason it's so much faster is because the developers at Intel behind it are professionals who lean on their profilers and measurements and really tuned the areas that mattered. They weren't changing code willy-nilly out of hunches and committing changes which made 0.000000001% improvements at the cost of significantly degrading maintainability. These were very precise optimizations applied in judicious hands -- they might have been microscopic in terms of focus but macroscopic in terms of effect.
Naturally with the real-time frame rate requirements of games, depending on how high-level or low-level you're working in with the game engine (even games made with UE 4 are often implemented at least partially in high-level script, but not, say, the most critical parts of the physics engine), micro-optimizations become a practical requirement in certain areas.
Another very basic area that surrounds us daily is image processing, like blurring high-res images in real-time and maybe doing other effects on them as part of a transition which we've probably all seen somewhere, perhaps even an OS effect. You can't necessarily implement such image operations from scratch looping through all the pixels of an image and expect such real-time results at matching frame rates. If it's CPU we're usually looking at SIMD and some micro-tuning, or we're looking at GPU shaders which tends to require a micro-level sort of mindset to write effectively.
If yes, as hardware improves should we expect higher level languages to take-over the gaming industry?
I rather doubt the hardware advancements alone could do that, because as the hardware advances, so too does the instructions and the technology (ex: physics on GPU), and techniques, and customer expectations for what they want to see, and competition, in ways that often have developers going low-level again, as with even the case of web developers now writing low-level GLSL shaders in WebGL (web development of this particular sort is arguably even lower-level than it was a decade or two ago, as GLSL is an extremely low-level C-like language, and I never would have guessed a decade or two ago that some web developers would embrace writing such low-level GPU shaders).
If there's going to be a way for performance-critical areas to move to higher-level languages, it's going to have to come more from the software and compilers and tools we have available as I see it. The problem to me in any foreseeable future isn't hardware not being powerful enough. It has more to do with how we can't find ways to most effectively talk to it every time it changes and advances without working our way back to its own language again. It is actually the rapid pace at which the hardware changes that makes high-level programming rather elusive for these areas as I see it, since if hypothetically our hardware stopped advancing out of the blue for the following decades, we might all gradually move to programming all of our games in the highest level languages using Unreal Engine 4 instead of constantly making new engines and kind of resetting that process every time the hardware advances.
Funnily these days, when I'm working in genuine performance-critical areas, I have to somewhat think more low-level than I started (even though I started in Borland Turbo C DOS era). Because back then the CPU cache was almost non-existent. It was mostly just DRAM and registers, which meant I could focus more on algorithmic complexity and write linked structures like trees in a very straightforward way without taking much hit to performance. These days the low-level details of the CPU cache dominates my thinking almost as much as the algorithm itself. And likewise we have multi-core machines which has to make us think about multithreading and atomics and mutexes and thread safety and concurrent data structures and so forth, which I'd say is, in many respects, a much lower-level focus (as in, not so humanly intuitive) than when I started.
Oddly that seems very true to me now. I think I'm more impacted by the underlying and low-level complexities and details of the hardware today than I was 30 years ago, trying my best to take off the nostalgia glasses. Of course we might have talked a little bit of assembly here and had to deal with some gory details like XMS/EMS. But for the most part I'd say there was less complexity and hardware and compiler awareness I required back then than I do today when I'm working in performance-critical areas. And that almost seems true of the entire industry if we put aside like writing if/else statements in a slightly more humanly readable way and consider how much people in general these days are thinking more about the lower-level details of hardware (from multiple cores to GPUs to SIMD to CPU cache and the internal details of how their compilers/interpreters/libraries work and so forth).
High Level != Less Efficient
Coming back to this question:
If yes, as hardware improves should we expect higher level languages to take-over the gaming industry?
To me it's not about hardware. It's about optimizers and tools. When I started people practically wrote all console games in assembly, and there was a genuine performance advantage then especially given the lack of quality compilers generating 6502.
As optimizing C compilers got smarter in their optimizations, then they started to reach a point where the higher-level code written in C was rivaling and occasionally even outperforming the code written by the finest assembly experts in many areas (though not always), and that made it so it was then a no-brainer to adopt C for at least the bulk of the coding for a game. And a similar shift gradually happened at some point with C++. The C++ adoption was slower since I think the productivity boost of going from assembly to C could probably reach unanimous agreement from gamedevs writing non-trivial games entirely in ASM as opposed to going from C to C++.
But these shifts didn't come from hardware becoming more powerful so much as the optimizers for these languages rendering going lower-level largely (although not always entirely, there are some obscure cases) obsolete.
If you can imagine a hypothetical scenario where we could write code in the highest-level code imaginable, with no concern about multithreading or GPUs or cache misses or anything like that (maybe not even specific data structures), and the optimizer was like artificial intelligence smart and could figure out the most efficient memory layouts rearranging and compacting our data, figure out it could use some GPU here and there, parallelize some code here and there, use some SIMD, maybe profile itself and keep further optimizing its IR as us humans do responding to profiler hotspots, and it did that in a way that beats the world's finest experts, then it'd be a no-brainer for even those working in the most performance-critical fields to adopt it... and that's an advancement coming from ridiculously smart optimizers, not faster hardware.
