It's really no different than building an application that can have multiple instances of itself.
From my experience, the biggest hangup is the persistence layer. Whether that's a configuration file or a call to a database, apps seem to get trapped into accessing a specific persistence object in a specific way which makes it all the more difficult to have multiple instances in place.
After that, you have to be a bit intelligent in how you lay things out. Visual indicators such as different backgrounds; title bar text; fonts; different account logins; different monitor locations; etc... all help make it clear which editor is primary / stable and which editor is in development.
It should be obvious, but this implies two separate code trees in the example you provided. One for the primary editor which is never changed. Another for the secondary editor which is under development.
And this approach holds up even to the extent of using the primary IDE to debug the development IDE as you mentioned in your question. When you're debugging the debugger capabilities of an IDE, then you'll need to feed a separate application into the development editor so it has something different to debug. In reality, that's not all that different from the tertiary project you would need to create in the secondary (development) editor in order to test the changes you made to the secondary editor.
It can be pretty easy to wrap your head around an axle with the meta aspects of building an IDE. The key is to clearly differentiate what each project is doing.
