Welcome to the world of legacy software development.
You have 100s of thousands, millions, 10s of millions of lines of code.
These lines of code are valuable, in that they produce a revenue stream and replacing them is infeasiable.
Your business model is based off of leveraging that code base. So your team is small, the code base is large. Adding features to is required to get people to buy a new version of your code, or to keep existing customers happy.
In a perfect world, your huge code base is unit tested up the wazoo. You don't live in a perfect world.
In a less perfect world, you have the budget to fix your technical debt -- break your code down into unit testable pieces, do exensive integration testing, and iterate.
This, however, is paying down debt without producing new features. Which doesn't match the business case of "reap profits from existing code, while modifying it in order to generate incentive to upgrade".
You could take huge chunks of code and rewrite it using more modern techniques. But everywhere you interact with the existing code you'll be exposing possible break points. That hack in the system that you got rid of actually compensated for a quirk in a subsystem you didn't rewrite. Always.
What you can do is act carefully. You can find some part of the code that you actually understand, and whose behavior and interaction with the rest of the system is well understood. You can modernize that, adding unit tests and making its behavior even clearer.
Then find the parts of the rest of the app that mainly interact with it, and attack them one at a time.
As you do so, you can improve the subsystem, adding features that the customers are willing to pay for.
In short, this is the art of the possible -- making changes without breaking things that provide a business case.
But this isn't your question. Your question is, "I am doing something that is huge, and likely to break stuff, and how do I follow best practices?"
When doing something huge, it is true that if you want to do it reliably you end up spending more effort tracking down bugs and fixing them than you do writing it. This is the general rule of software development: writing stuff is easy, making it work flawlessly is hard.
You probably have a business case hanging over your head, where you have promised to some stakeholder that this massive change goes in. And it is "done", so you get pushback on saying "no, this isn't done, it just looks like it".
If you have the power and the budget, actually spend the effort generating confidence that the change works, or simply reject the change. This is going to be a matter of degree, not kind.
If you don't have that much power, but still have some, try to insist that the new system is unit testable. If you rewrite some subsystem, insist that the new subsystem is composed of small parts with well specified behavior and unit tests around them.
Then there is the worst case. You go deeper into debt. You borrow against the future of the program by having more code that is fragile and more bugs in order to get the feature out now, and damn the consequences. You do sweep-based QA to find the worst issues, and ignore the rest. This is actually sometimes the right answer from the perspective of the business, as it is cheapest now. Going into debt to generate profits is a valid business strategy, especially if clearing the debt via bankruptcy (abandoning the code) is on the table.
A large problem is that rarely are the incentives of the company owners aligned with the decision makers and the programmers. There tends to be lots of pressure to 'deliver', and doing so by generating nearly invisible (to your superiors) technical debt is a great short and sometimes medium-term strategy. Even if your superiors/stakeholders would be best served by not creating all that debt.