How to prevent the vicious circle of testing via customer

Question

I recently asked this question and this comment and made me ponder the whole process of releasing an update to the customers. The next update for our software is planned and the previous time it went horribly wrong because of regression, bad testing, etc...

This time we want to prevent the scenario that we test via our customer and made some unit tests and other minor improvements. However, there is not enough time to make unit tests for all the features that lack unit tests or the features that need to be build yet. So the answer for this question 'Make more unit tests' is not fitting because of the lack of time, infrastructure etc...

Every bug and feature will be tested/reviewed by a different developer, but that won't be done perfectly of course (we're still human :)).

My question is: Once we released the update, how do we prevent the vicious circle of testing via customers while not be able to add more unit tests?

---

The previous time we tried our best to test the software before and after release. After the deadline we were still searching and fixing bugs but the customer would also find some of these. So we post an update as soon as we were confident that the patch was increasing quality. We were so busy with finding bugs and fixing them that we weren't spending any time on creating unit tests.

This time we don't have a solid plan yet, hence the question. As stated before we did added some unit tests, but surely not enough. We are also looking for new colleagues but they are not easy to come by and management has rather high standards. We do have quite some candidates and it looks like we do get at least one new colleagues very soon, but it still takes time to get him/her up and running. The first thing we put the new colleague on will be creating unit tests. So the dependencies there are experience, skills and the time he/she can join the time.

Another problem that comes in hand with the lack of developers is that we are so focused on getting the bugs and features fixed that we don't have much time to spend on thinking about 'how we can improve the update process'. (I ask these questions in my own spare time).

---

Summary of steps I will try to take:

• Try to get a tester before the release is done
• Add unit tests while creating the features, if there isn't enough time, schedule them for after the release.
• After releasing, if there is any regression, create unit tests when fixing the bugs.
• Release more often in smaller chunks after the release, making it easier to release features and repair bugs with a minor priority.

Answer 1

There is more to testing than unit testing. You need to test the new features and bug fixes you created manually in a testing environment that mirrors a real-world installation as best as possible. Having one of the other developers 'review' changes is not an appropriate level of testing for two reasons. First, that implies they'll give it a quick look and then say 'sure, it works'. Two, developers use software differently than end users - particularly software they wrote - so they may not catch problems that an end user will immediately see. Ideally you need someone who is not a developer to run through a checklist of tests to confirm your change is working correctly before a customer ever gets their hands on it. I suspect if you were testing thoroughly in this manner, you would never have been getting 3 patches a week out.

Expanding this a bit based on your other question...

You also need to get into a release schedule. Multiple releases a week is too high a pace to give you the time necessary to test a release thoroughly. This is especially true if you are seeing a problem of your patches introducing new bugs. If you are concerned that a customer will be held up waiting for a fix if a bug is introduced in a patch, better to give them an easy way to revert to the previous working version and have them wait until the next scheduled release date so you are not rushing things out the door. My advice to you is to attempt a 1 month release schedule at first. Company management may resist this because they want to appear 'responsive' to customers. Having seen that attitude in companies I have worked for, I assure you, you are not doing the customer any favors by giving them improperly tested updates. Many non-technical decision makers will push back against doing this because they don't undestand the problem. Unfortunately all you can do is lead a horse to water... you can't make it drink. Its your professional responsibility to tell them the wise course of action, and not yours if they won't follow it.

Answer 2

The answer is more tests. Unfortunately, that isn't the answer you want to hear, but it really is the only solution. As you noted, the last roll-out went disastrously, which is most detrimental to customer satisfaction. It would be better to build a solid product with fewer new features than to build a whole bunch of buggy features on a buggy base. "This is a great new feature, but unfortunately it deleted all my data!"

For the as-yet-unwritten features, definitely write tests (and my opinion would be to at least do test-first programming if not full TDD). These are greenfields, so you can make the efforts to make them good from the start. Prioritize the new features, and do the most important first. Less important features will be less missed for a release if they work well.

For the already-written features, prioritize those into a list, and work your way through that as well. Obviously your progress will be dependent on how important each feature is and how important developing new features is, but make time to work through this list as you can.

For the bugs, especially for the bugs, write tests when you fix them. The test can just be a guard against what you fixed to start, but at least you know you won't make that mistake again.

---

I do have personal experience with this type of workflow, and I can say that I have had positive results. It can be a real slog, but it works out for the better.

Answer 3

Focus on just doing a good job with your tasks.

Work on being a better developer. Solve your own imperfections before you worry about that of the company.

Create a reputation in the company as a person who can be trusted with an assignment. Test your own work. Write unit tests for your changes. Demonstrate the behavior of a professional developer that creates quality software.

When time comes they will ask you "How do you do it? Everyone else is far less effective".

Once you have their ear. They will be willing to listen, but first you have to demonstrate that quality is possible.

The first step in introducing change in the people around you. Is to first change yourself. Quality software starts first with the individual.

The benefit in changing how you work is that it will attract like minded people. One advantage is that it often attracts people from outside the company. While your current employer may not embrace the new you and adopt an attitude of quality control. There are other businesses that will open doors for you. You will find that you have more in common with their goals, then the goals of your current employer.

Answer 4

You need to test intelligently. For now, make a dependency tree of the files/features/components in the system. Mark the ones that have had work done on them, test those and any other files/features/components that take dependencies on them.

What this allows you to do is write, or perform, tests surgically. You need:

Testers

From what I am hearing, you don't have a lot of test code. You need bodies in seats, that are not developers (ideally people closest to what a customer is, but with test experience/skills). These people should hammer the features, that you've identified need testing, with all kinds of well though out test cases. Test case design/selection is a rigorous task in and of itself (all to many times have I seen redundant tests in test plans).

Test Code

Again, unit tests can be helpful, but they are often too granular for the situation you're in. You need integration tests. Write rigorous test cases for the inputs/outputs for your customer facing components/services. You'll get better test coverage in a short timespan with targeted, repeatable, integration tests. In the short term, try not to overlap integration test code with test cases being performed by testers unless it is a critical component.

Answer 5

There is much more to testing than unit testing, regression testing, or exploratory testing. Unit testing only has so much value, writing tests that the programmer knows will pass now, is only useful to populate a regression suite or to help guide you before the code is written as in Test Driven Development.

There is a great diagram on Lisa Crispin's site that talks about the "Agile test quadrants". It talks about four quadrants split between

Business Facing vs Technology Facing

(Functional Test etc.) / (Scenarios etc.)

Supporting the Team vs Critiquing the Product

(Unit tests etc.) / (ility testing eg scalability, usability etc.)

There is a good book, Agile Testing that talks a lot about all the other types of testing apart form Unit testing. It is not easy to do this, especially under time pressures and it is hard for all programmers, and very hard for some, to think in the analytical way necessary to test using this approach but it is the best way I have found so far.

By analysing each new feature by at least considering each quadrant you can decide quickly what needs to be done and what can be left out or deferred.

To be able to decide what needs to be done you really need a good idea of the state of your product. What areas are a concern? What features do you know are not as reliable as the rest of the product? You also need a way to gain confidence in a candidate for release. How long does it take between writing a new functions and the developer being able to see if it has broken anything else? How long does it take to be able to get a build up and running and see if the product works as expected, not just seeing that the function returns the 'correct' value, but interacting with the system as the user will and seeing if it acts as I expect?

The other non-testing part of testing is that when you are considering the other types of testing (not unit testing), are you validating the results against what the user is expecting? Are you doing hall way usability testing (getting some one not in your team, who doesn't know the code or what the feature is 'supposed' to do to try interacting with the system to see if it makes sense)? Do your User Stories (or high level sets of tasks to complete) talk about the end user and what they expect to happen and why they are using the product in this way, what are they trying to achieve, what do they care about? Are you checking that what you are building is what the user needs or just some engineered approximation of a feature set that bares a resemblance to what they asked for?

If an organisation thinks that testing (the department / team / individuals / or developers that are doing it this week) is doing such a poor job that it has become a problem for the business (sometimes it is appropriate to put out buggy software, professional software developers don't like it, it feels like someone scratching a blackboard, but that doesn't mean it's wrong) then I would guess that most of the time it isn't a problem that can be solved by any extra testing, test strategy, or 'agile' practices but it is the development practices that need looking at, the attitudes or the bureaucracy and process around requirements and developing and people not realising the consequences of their current practices.

Answer 6

Pain Reduction

One approach that can work and hasn't yet been mentioned here is to lower the cost of failure. One contributing factor to the magnitude of the disaster of your last release is the pain caused to the customer. If you make it trivial to roll back a release version - hopefully in a self-serving manner so customers really are minimally inconvenienced, then you can release with more confidence that your software release won't ruin anyone's day.

Now, this obviously adds more complexity to your product, costing its own amount of time and budget to implement and introducing another channel for bugs. Especially if your app is responsible for managing large amounts of customer data, this could be more trouble than it is worth. (How to tell? If you rely on a file system store with no watching or auto-load hooks, you're probably safe. If you install a database on a client's machine, you need to be very cautious.)

However, if you have a simple desktop application, there are plenty of ways to make version swapping painless. You could also take the precaution of forcing customers to backup their data before updating. Or if part of your new features include a new file format, you could auto-convert files on first load and disallow the user from overwriting their previous version files.

This approach is even more practical if you host your application on the web. In that case, a browser is essentially downloading and re-installing your software for your every time the customer loads a page.

Testing Priority

The other way to quickly build out your test suite with high-priority tests is to require every bug fix to include a test that fails if the bug is not fixed and passes when it does. Cbojar mentioned this in their answer, but I wanted to expound upon the many reasons this is a good idea.

This approach is often popular with management, because it doesn't require an up-front block of test writing and it addresses your customers' problems in real time. So when a customer runs into a bug and complains loudly, you get to say "Yes, we've just fixed that bug for you. And we're confident it will never happen again." And that's an extra incentive to writing the test and writing it thoroughly and well, because you know a customer is depending on it.