Yes, you would have Factory of Factories. The key point is that this either class or method should be in topmost library/application. Only smallest possible part of the application should know about this FoF. It could very well be a simple method in launcher of the application that is executed before the main loop of the application executes.
If you were tempted to create interface for this FoF, then that would be indication that your creation scheme might use redesign.
Edit : It seems we need to clarify what I mean by Factory (actually Abstract factory, but I don't consider "a Factory" as pattern. Abstract Factory pattern is solution to a situation, where some piece of code needs to create an instance, but the piece of code itself doesn't have enough information to decide which instance to create. That is why the client exposes an abstraction and someone who has information to create that instance realizes that abstraction.
If you want to talk about design, you MUST ALWAYS tell us how is it going to be used and who has what information.