Testing framework upgrades

Question

Due to many vulnerabilities disclosures, we had to upgrade our core framework, on which large parts of our system depend. (In our case, it was Spring framework version from 4.3.16., but that is just an example, probably less important.)

We fixed some of the backward compatibility issues in the new version.
(Like : new methods in interfaces which was implemented in our project)

How to test if this doesn't cause any problem ?
I am wondering if I could follow any approach to make sure it doesn't cause any other issues to be detected later during runtime.

Also, this might be a common concern for any library upgrade. Are there any practices or procedures which should be done to eliminate such concerns?

Answer 1

You test. And then you test some more. And then you test a bit more.

That's obviously a bit general, but it's the answer. In a bit more detail:

• You make sure your unit tests pass (failing unit tests should always block the build)
• You make sure your integration tests pass
• You make sure your automated end-to-end tests pass
• You do some experimental testing on the system. Is anything wrong?
• You run the formal regression test plan.

Repeat across your various environments (development, internal QA, UAT, whatever your process involves) until such point as you have enough confidence in the change.

Finally, deploy to production.

Answer 2

Too many times I see this process for upgrading frameworks:

1. Halt all new feature development
2. Upgrade framework
3. Test manually
4. Keep testing and fixing
5. Keep testing and fixing
6. Deploy to a pre-production environment... and keep testing and fixing
7. Repeat for each environment
8. Deploy to production
9. Pray to whatever deity you believe in
10. Keep testing and fixing

Step 2 should actually be step #3. Step 1 should be entirely unnecessary. If you need to do step #1, you are doing it wrong.

Step 1 should be start with unit and automated test coverage.

I was part of an agile team that not only was able to do a non compatible framework upgrade, but also a non compatible upgrade to the language itself. It was a large Ruby on Rails application upgrading to Ruby 1.9. So basically every framework, library and dependency we had was also upgraded or replaced.

In 4 weeks. In 4 environments, including production. And we delivered 4 major new features as well.

This is not a way to say "Ruby is better!" rather, it is a testament to the amount of unit and automated test coverage the application had. We had 1,200+ unit tests, and around 600 cucumber tests. It is also a testament to a tightly knit team with great communication.

So, my amended list of steps for upgrading a framework, or major piece of your software infrastructure:

1. Write lots of unit tests. Refactor code if necessary so you can write unit tests for the core logic of your application

2. Create automated functional tests. Not everything can be tested with a unit test, but a full functional test through the UI is the next best thing. Write lots of these.

3. Repeat steps 1 and 2.

4. Upgrade the framework.

5. Get the unit tests to pass.

6. Get the functional tests to pass.

7. Keep track of the breaking changes you need to make. E-mail your team about these changes and how they can fix work in progress.

8. Team code review. Have them ask questions. Answer them in great detail.

9. Check code in, and make sure your continuous integration build is passing.

10. Assist teammates with merges and bug fixing for work in progress.

11. Deploy to a pre-production environment for manual testing. Have a manual test plan already in place.

12. Deploy to the next pre-production environment, followed with some more testing.

13. Deploy to production.

The key difference between the two approaches is when you work out the kinks. With approach #1 you encounter most issues during manual testing. With approach #2 you encounter them during development, where they are easier to track down and fix.