2

Our company still uses a lot of manual testing of software (and this will not change in the coming years). We try to improve our build and deployment process by using a deployment pipeline which handles both the automatic and the manual tests.

The idea is to present the user a list of tests for which he approves that they were executed. The deployment pipeline waits until this approval was given.

The main problem in this sort of formalised deployment pipeline is that it is not possible to redo all manual tests after a bugfix. In a completely automatic deployment pipeline, you would fix your code, create a new version of your artifact and send it (again) through all tests. We do not have the resources to repeat all manual tests, but rather we would only manually test those parts of the software that are "probably" affected by the bugfix.

What would be a good way to handle this in a deployment pipeline?

  • Who executes the manual tests? Is it the same people who are building the software or is it a subset of people (such as a QA organization or developers who are specifically assigned to manual test execution)? – Thomas Owens Oct 18 '17 at 13:12
  • Most of the tests are executed by non-developers. We develop software for different departments of our company - these departments have designated people for testing. – J. Fabian Meier Oct 18 '17 at 13:18
  • Do you have tool support for your deployment pipeline, like some kind of CI server? – Thomas Owens Oct 18 '17 at 13:21
  • We are working on this, but probably Jenkins will be the tool. – J. Fabian Meier Oct 18 '17 at 13:23
3

I can't give you an answer in any context of "who's fault would it be if problems were to arise in production", as there is only one best practice to minimize errors and that would be to simply retest everything anyway.

Testing areas "probably" affected by the bug fix is a reasonable compromise, though only the person who made the bug fix could know which areas are affected. You could possibly associate tags with each test and then say, test everything containing tags "dao" and "orders". This would give you a little flexibility in that you can departmentalize your program and therefore your tests so you can quickly and fairly accurately test these areas affected when a bug fix is made. It might be worth your time to create a technical document for your fellow developers to clarify which areas in the program correspond to which tags.

Hope that helps!

3

Step one should be to deploy a CI server, such as Jenkins. There may be some development-side work to improve this - have a good branching strategy and target your CI server at the appropriate branches, get your build process automated, incorporate any automated tests that you have into your CI pipeline.

One of the things that many (if not all) CI servers allow you to do is to have a manual promotion process. With appropriate configuration and scripting, you can incorporate both automated steps (automated testing, static analysis) as well as manual approval. If you kick off automated tests on a regular basis, you can identify every successful build. Then, with appropriate permissioning, you can allow specific users to use the web interface to tag and deploy to various levels of integration and test environments. Eventually, you can even push out production builds of your software by people promoting builds that pass each gate.

The thing that I'm not sure about is using the tool to enforce that test cases have been run. There may be appropriate plugins, but this may be a manual process. If you are using a test management tool, your CI server may plug into that and only allow promotion if a specific test suite has been marked as passing by people in a tool. It's not necessarily foolproof - people could mark as passing without actually running the tests. But I do think that you need to trust your people at the end of the day to do the right thing.

Overall, I think your strategy of choosing test cases to run based on the code changes is the right thing to do. However, I would also consider trying to invest in improving some levels of automated testing to try to find problems before it hits manual testing, but this would need to be a long-term investment and may require either training people on new skills or hiring people with certain skills.

0

I would try and simplify this a bit, instead of create a list of tests to run...

Try to create a "Baseline" of tests to run after a build and deploy. This can be a mixture of manual and automated tests. It should include a basic "Smoke Test" to validate all major working parts of the application. Run the baseline and if it passes, then do the specific defect tests. If those pass, the defect can be closed.

At a set interval do a regression test run, maybe 1x week, 1x month, etc. to catch any issues that were indirectly caused by defect fixes. Manual regression is expensive so we can not do it every time a build is done, but it is needed to ensure any updates did not break any existing functionality which can sometimes happen.

The more tests are automated, the faster one can validate the changes and the faster any fallout can be detected. If everything is automated one can make a change and validate it very quickly without a lot of manual effort so the speed and agility of your software increases dramatically.

The baseline can altered as needed. Baseline tests should only take a short time to validate if correct, like < 1 day, if they were automated maybe just a few minutes at most. Baseline should include major functionality but not cover everything.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.