Keep optimisations local, make them obvious, document them well and make it easy to compare the optimised versions with each other and with the unoptimised version, both in terms of source code and run-time performance.
If such optimisations really are that important to your product, then you need to know not only why the optimisations were useful before, but also provide enough information to help developers know whether they will be useful in the future.
Ideally, you need to enshrine performance testing into your build process, so you find out when new technologies invalidate old optimisations.
The First Rule of Program Optimisation: Don't do it.
The Second Rule of Program Optimisation (for experts only!): Don't do it yet."
— Michael A. Jackson
In order to know whether now is the time requires benchmarking and testing.
As you mention, the biggest problem with highly optimised code is that it is difficult to maintain so, as far as possible, you need to keep the optimised portions separate from the unoptimised portions. Whether you do this through compile time linking, runtime virtual function calls or something in between shouldn't matter. What should matter is that when you run your tests, you want to be able to test against all of the versions you are currently interested in.
I would be inclined to build a system in such a way that the basic unoptimised version of the production code could always be used to understand the intent of the code, then build different optimised modules alongside this containing the optimised version or versions, explicitly documenting wherever the optimised version differs from the base-line. When you run your tests (unit and integration), you run it on the unoptimised version and on all current optimised modules.
For instance, lets say you have a Fast Fourier Transform function. Maybe you have a basic, algorithmic implementation in
fft.c and tests in
Then along comes the Pentium and you decide to implement fixed point version in
fft_mmx.c using MMX instructions. Later the pentium 3 comes along and you decide to add a version which uses Streaming SIMD Extensions in
Now you want to add CUDA, so you add
fft_cuda.c, but find that with the test dataset that you've been using for years, the CUDA version is slower than the SSE version! You do some analysis and end up adding a dataset that's 100 times bigger and you get the speed-up you expect, but now you know that the set-up time for using the CUDA version is significant and that with small datasets you should use an algorithm without that set-up cost.
In each of these cases you are implementing the same algorithm, all should behave in the same way, but will run with differing efficiencies and speeds on different architectures (if they will run at all). From the code point of view though, you can compare any pair of source files to find out why the same interface is implemented in different ways and usually, the easiest way will be to refer back to the original unoptimised version.
All of the same goes for a OOP implementation where a base class which implements the unoptimised algorithm, and derived classes implement different optimisations.
The important thing is to keep the same things which are the same, so that the differences are obvious.