I've attempted pair-programming several times, including in an organization that (briefly) considered rolling it out as a mandatory process for all engineers (you can guess how well that idea panned out). Personally, I despised it.
The reasons I list below are only my subjective experiences, and I cannot 'measure' their impact in concrete terms. But here they are all the same:
1 - Having a 'navigator' and a 'driver' only helps if the former is vocal and the latter will listen.
We've all met developers who are stubborn, zealous about some theoretical concern or pathologically unable - psychologically - to 'throw away' old work when someone suggests a problem with it. And we all know individuals too timid or diffident to raise concerns or suggest corner cases.
When these kinds of developers are paired, the navigator quickly takes a passive role, and what you end up with is sole-programming with an automatic code review. This is a monumental waste of resources.
2 - Pairing prevents creativity.
Contrary to what was formerly felt about the value of 'group brainstorming', the consensus these days is that creative knowledge work requires independence and autonomy. When you're working alone, you can quickly hack together some crazy idea to see if it's actually feasible. You can wordlessly assemble some strange prototype, and if you fail, it doesn't matter, because no-one knows.
Compare that to pairing: when I want to try out some new concept, I have to convince my partner, talk them through the implementation, step-by-step, and hope that they won't judge me if it fails. That kind of environment is toxic to creating new ideas.
3 - Lowest common denominator design.
When a pair cannot spin up new ideas, as above, or when the individuals cannot agree on some fundamental principle of how a feature should be designed, what comes out is a muddled design that attempts to compromise and satisfies no-one.
If you pair a developer who builds wonderful, eloquent, skyward functional programming abstractions with a fast-and-dirty performance freak, the code they'll together produce will typically be neither terribly elegant nor particularly fast.
4 - Lack of autonomy and violent transparency.
Violent transparency is a phrase I've plucked from a moderately famous (and quite controversial) polemic against the Scrum methodology. It describes the way that some organisations infantilise developers and treat them with the suspicion normally reserved for non-professional workers.
Whatever you think about the 'harms' of making developers' work entirely transparent (and you may not agree it is actually a harm), many individuals value their autonomy and their ability to work alone, trusted to do the right thing. It's an important psychological need, and to force developers into pairing (as I've seen happen in at least one shop) will leave employees dismayed, upset and alienated.
5 - Some developers just don't play nicely in pairs.
Some people won't or can't conduct themselves appropriately in a paired environment. They may have bad hygiene, poor working habits, an abrasive personality, a 'loud' and 'intense' manner, or a whole host of other attributes that make them fine individual workers, but poor pair programmers.
Can you solve this? Not really. Changing personal behaviour is tough. A pair-programming shop needs to be very careful about hiring and invest a lot of time into seeing how someone works and whether they'll be able to work well with their peers. Discriminating harder on personality, however, means that hiring will take longer unless you loosen your standards on skills and expertise.