What is the safest way to deploy a lot of changes at once to a legacy, monolithic application with no test coverage?

Question

My team have recently inherited a very poorly written, business-critical, monolithic LAMP application with the goal of guaranteeing reasonably uptime and scalability targets in just a few weeks. Our deadline is somewhat fixed because it depends on user volumes over the peak season, which we forecast will start picking up around Black Friday.

The monolith is currently deployed to a single, "snowflake" EC2 in AWS. Deployments to this instance are entirely manual. The biggest operational risk we're currently trying to mitigate with this instance is that it currently has a dynamic IP attached, which the cloud provider would likely recover if the instance rebooted due to a server crash or maintenance work. We've spoken to AWS about it and they've confirmed they would not be able to give the IP back to us. This would break dozens of integrations that are (wrongly) currently pointing to this ephemeral IP address start failing. By some miracle though, the VM has been stable enough not to crash once so this situation has never happened before but that doesn't mean the risk of it happening is not there, especially during peak traffic.

The monolith is very tightly coupled to the Production environment in which it runs, as it has a ton of hardcoded references to its own Production ephemeral IP address. Moreover, all of its application config is hardcoded and duplicated across every dozens of file. All of these settings assume they're running in a Production environment too.

We want to make the minimum amount of changes to the application code to allow us to replace its dynamic IP with a reserved, static one. However, because the monolith is not modular at all and there is little to no code reuse in it, we've basically had to change hundreds of files just to be able to fetch all the application settings from a single config file. This has allowed us to deploy a version of the monolith with Test settings to a Test environment that we're able to run manual tests against.

Even though each individual change we've made is very small, we have basically touched almost every source file in the codebase. The people that wrote this legacy application and therefore had any good understanding as to how it works no longer work at our company. With no support from domain experts and no automated test coverage whatsoever, we're looking at a pretty risky, time-sensitive deployment that we know needs to happen as things could get pretty ugly for us if the instance reboots or dies for whatever reason and we lose its IP.

So the main question facing us now is, what would be the safest way to make this deployment happen as safely as the circumstances allow? We've also ruled out accomplishing this through multiple deployments because of how tightly coupled and messy the code is.

I'm not looking for any silver bullets here. Any ideas or suggestions would be more than welcome.

Answer 1

Since you don’t have any automated testing that would find bugs, the next best thing is to make changes while introducing the smallest possible number of new bugs.

Writing good code carefully and code reviews is one way. There’s a better way: Take two good and reliable developers. Let them agree what is to be changed and exactly where. Let them both make the changes independently. And then compare the results. After two or three practice runs they should be able to create identical code except for errors that are promptly fixed.

A major advantage to a pure code review: Code review won’t find places where a change was missed. Two independent developers, at least one shouldn’t miss a needed change. Important is that they don’t agree beforehand about all the changes they need to make, because that would create the risk that both miss a change.

Answer 2

When you don't have any automated tests and cannot easily write them, all you can do is testing the system manually. You will need a test environment which is very similar to your production environment. If you cannot afford this (or even when you can afford this, but have no chance to simulate the system with realistic usage scenarios in your test environment), the best thing you can do is to have a deployment strategy where you can quickly switch back to a prior version in production after you start observing serious issues (which is not unlikely after your first deployments).

First and foremost you should care for preventing data loss, of course, even when you have to switch back from the newest version to the latest stable version. It sounds like your new version does not require a new database schema version, hence I guess that should not be much of a problem in your case (still you should think about what the business risk is when your deployment has to be rollbacked).

So in short - hope for the best, but be prepared for the worst!