Manual testing in the deployment pipeline

Question

Our company still uses a lot of manual testing of software (and this will not change in the coming years). We try to improve our build and deployment process by using a deployment pipeline which handles both the automatic and the manual tests.

The idea is to present the user a list of tests for which he approves that they were executed. The deployment pipeline waits until this approval was given.

The main problem in this sort of formalised deployment pipeline is that it is not possible to redo all manual tests after a bugfix. In a completely automatic deployment pipeline, you would fix your code, create a new version of your artifact and send it (again) through all tests. We do not have the resources to repeat all manual tests, but rather we would only manually test those parts of the software that are "probably" affected by the bugfix.

What would be a good way to handle this in a deployment pipeline?

Answer 1

I can't give you an answer in any context of "who's fault would it be if problems were to arise in production", as there is only one best practice to minimize errors and that would be to simply retest everything anyway.

Testing areas "probably" affected by the bug fix is a reasonable compromise, though only the person who made the bug fix could know which areas are affected. You could possibly associate tags with each test and then say, test everything containing tags "dao" and "orders". This would give you a little flexibility in that you can departmentalize your program and therefore your tests so you can quickly and fairly accurately test these areas affected when a bug fix is made. It might be worth your time to create a technical document for your fellow developers to clarify which areas in the program correspond to which tags.

Hope that helps!

Answer 2

Step one should be to deploy a CI server, such as Jenkins. There may be some development-side work to improve this - have a good branching strategy and target your CI server at the appropriate branches, get your build process automated, incorporate any automated tests that you have into your CI pipeline.

One of the things that many (if not all) CI servers allow you to do is to have a manual promotion process. With appropriate configuration and scripting, you can incorporate both automated steps (automated testing, static analysis) as well as manual approval. If you kick off automated tests on a regular basis, you can identify every successful build. Then, with appropriate permissioning, you can allow specific users to use the web interface to tag and deploy to various levels of integration and test environments. Eventually, you can even push out production builds of your software by people promoting builds that pass each gate.

The thing that I'm not sure about is using the tool to enforce that test cases have been run. There may be appropriate plugins, but this may be a manual process. If you are using a test management tool, your CI server may plug into that and only allow promotion if a specific test suite has been marked as passing by people in a tool. It's not necessarily foolproof - people could mark as passing without actually running the tests. But I do think that you need to trust your people at the end of the day to do the right thing.

Overall, I think your strategy of choosing test cases to run based on the code changes is the right thing to do. However, I would also consider trying to invest in improving some levels of automated testing to try to find problems before it hits manual testing, but this would need to be a long-term investment and may require either training people on new skills or hiring people with certain skills.

Answer 3

I would try and simplify this a bit, instead of create a list of tests to run...

Try to create a "Baseline" of tests to run after a build and deploy. This can be a mixture of manual and automated tests. It should include a basic "Smoke Test" to validate all major working parts of the application. Run the baseline and if it passes, then do the specific defect tests. If those pass, the defect can be closed.

At a set interval do a regression test run, maybe 1x week, 1x month, etc. to catch any issues that were indirectly caused by defect fixes. Manual regression is expensive so we can not do it every time a build is done, but it is needed to ensure any updates did not break any existing functionality which can sometimes happen.

The more tests are automated, the faster one can validate the changes and the faster any fallout can be detected. If everything is automated one can make a change and validate it very quickly without a lot of manual effort so the speed and agility of your software increases dramatically.

The baseline can altered as needed. Baseline tests should only take a short time to validate if correct, like < 1 day, if they were automated maybe just a few minutes at most. Baseline should include major functionality but not cover everything.