2

I've worked on a few codebases without a great automated test suite, where changes that concern the whole platform have to be thoroughly tested by developers and there would be a high risk that a degradation is introduced with each release. How do you prevent this from happening?

The less than perfect answers from my experience are:

  • Test thoroughly yourself - completely unsustainable on large projects
  • Organise group testing sessions - can be effective, but a bit chaotic and a pain in the ass for everyone involved
  • Manual release QA tests - soul sucking affair where some poor sod has to trawl through a spreadsheet of manual tests, limited by their knowledge of the platform and existing bugs

What would be yours, save for improving the testing suite or hiring people whose job is QA?

2
  • 1
    Very careful and through code reviews.
    – JacquesB
    May 19, 2021 at 18:45
  • Unfortunately the two things you don't want to do (improve the test suite and hire a QA tester) are precisely the things you need to do. In fact, I would argue for hiring a dedicated QA tester even if you have good test automation. Testers think differently than developers, and will catch errors that developers don't think about. May 20, 2021 at 16:15

2 Answers 2

6

Tests prevent regressions. There are other tools which would prevent regressions, but unfortunately they are either less effective (example: code reviews), or extremely expensive (example: formal proofs), which leaves us with tests as the best option.

So you find yourself in front of a codebase which should be tested, but is not tested yet, or not enough, and you have to make a change. The solution to prevent regressions is to progressively increase the test coverage.

When you touch a small part of the codebase, check if it's tested. If you are confident enough that most if not all the logic is tested, that's great. If not, start by adding tests. The great thing is while you do it, you will:

  • Understand better the part that you wanted to modify.
  • Have a better view of how well this part of the code matches the business requirements.
  • Find a bunch of errors in the logic of the code, such as the branches which can never be reached.
  • Get a clear picture of the change you were expected to do.
  • Imagine some possible refactoring to do.

The elements in bold would reduce the risk of causing regressions by a change. Most regressions come from changes where developers misunderstood the code they were changing, or misunderstood the change itself.

Once you tested it:

  1. Write the test which will check for the change you're about to do.
  2. Do your change.
  3. Re-run the tests.
  4. Refactor.
  5. Re-run the tests.
  6. Release.

If you're not under time pressure:

  • Make additional effort to refactor more, if needed. Sometimes, there are opportunities to simplify the code tremendously, eventually replacing dozens of lines of code by a call to a method, or removing a duplication, or making a challenging algorithm easy enough for a junior programmer to understand. Since the code is tested now, you can refactor aggressively, without the risk of creating regressions while doing so.
  • If during testing, you noticed discrepancies between code and business requirements, go see the product owner.
  • If you still have free time, test code that surrounds the piece you already tested, especially if it allows to expand the refactoring further on.

Make sure you also read I've inherited 200K lines of spaghetti code — what now?

7
  • Thanks for the comment, especially the link is extremely insightful. I largely agree with your assertion about the benefits of tests - the issue is that sometimes testing is not an option for x or y reason (usually pressure from product or technical blockers) - it is for these cases that I am looking for a short term fix so that people's lives are not spent in misery until the team can get around to writing tests.
    – styke
    May 19, 2021 at 21:53
  • 3
    You're in a plane and your flight is delayed because of a safety issue. You're in a hurry: in three hours, you have to be in LA for the important meeting. Would you prefer the pilots to take off and see if the plane crashes or not, or you prefer to wait a bit until the safety issue is resolved? Software development is exactly the same. It belongs to you to explain to non-technical persons that you can release soon and the app may crash, or you may take the time you need and do quality work. If the stakeholders decide that they want to take the risk, so be it. But at least be clear with them. May 19, 2021 at 22:02
  • Also see an article I wrote on the subject. May 19, 2021 at 23:39
  • @styke: To sum up Arseni's first comment, never tell them "no" but always inform them of the consequences. And if you aren't comfortable with the consequences, you can always find another job. Don't put yourself in a position to be their scapegoat. May 20, 2021 at 16:19
  • I wouldn't even bother informing the stakeholders about the consequences, nor give them an opportunity to decide whether I should test my code or not, unless I have a very good reason to think that taking risks by skipping tests could make business sense. It doesn't belong to non-technical persons to decide how I should do my job: otherwise, they would be writing code in my place. It is essentially exactly the same thing as a pilot who would delay a flight for safety reasons, disregarding what the passengers want or think being the best option for them. May 20, 2021 at 23:54
1

Writing tests really is the way to go. but assuming you can't I would go with "more logging"

If you are logging all your errors then you can tell if a release increases or decreases the number of errors of various types.

If a feature release decreases errors, you are good. If it increases you have introduced a new problem.

Coupled with a canary release, where you release a new version to a subset of users you should at least be able to gauge if the new release is better than the last, even if you can't tell if it's working correctly or not.

1
  • That's a very good shout
    – styke
    May 19, 2021 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.