What can be done to decrease the number of live issues with applications?

Question

First off I have seen this post which is slightly similar to my question. :

What can you do to decrease the number of deployment bugs of a live website?

Let me layout the situation for you. The team of programmers that I belong to have metrics associated with our code. Over the last several months our errors in our live system have increased by a large amount. We require that our updates to applications be tested by at least one other programmer prior to going live. I personally am completely against this as I think that applications should be tested by end users as end users are much better testers than programmers, I am not against programmers testing, obviously programmers need to test code, but they are most of the times too close to the code. The reason I specify that I think end users should test in our scenario is due to the fact that we don't have business analysts, we just have programmers. I come from a background where BAs took care of all the testing once programmers checked off it was ready to go live.

We do have a staging environment in place that is a clone of the live environment that we use to ensure that we don't have issues between development and live environments this does catch some bugs.

We don't do end user testing really at all, I should say we don't really have anyone testing our code except programmers, which I think gets us into this mess (Ideally, we would have BAs or QA or professional testers test). We don't have a QA team or anything of that nature. We don't have test cases for our projects that are fully laid out.

Ok, I am just a peon programmer at the bottom of the rung, but I am probably more tired of these issues than the managers complaining about them. So, I don't have the ability to tell them you are doing it all wrong.....I have tried gentle pushes in the correct direction.

Any advice or suggestions on how to alleviate this issue ?

Answer 1

We require that our updates to applications be tested by at least one other programmer prior to going live.

That's a start, but you are writing here because you have proven this to be insufficient.

Applications should be tested by end users

We used to joke that we'd let end-users test our code. The point of testing is to find bugs before the end-users do. That way, end-users have a good experience, and recommend, or buy more of the software.

not [tested by] programmers

What's wrong with programmers testing? I think it's an important part of being a good programmer. Some schools of thought recommend that programmers write tests before even writing code! I think the problem that you are getting at is only having programmers do testing.

staging environment in place that is a clone of the live environment

This is very good. But if it is truly a clone, then the learning you gain from installing from development to staging is not getting transferred from staging to production. One way or another, you aren't getting the full benefit of this system. Do you do your testing in the staging environment?

We don't do end user testing really at all.

End user testing is usually about acceptance of new features, ease of use, and meeting the users needs. Not about finding bugs. End-users finding bugs = lost customers.

We don't have a QA team or anything of that nature. We don't have test cases for our projects that are fully laid out.

Eek! This is the root cause of your problems. A QA team would be fantastic, but even one eager and capable full-time tester would be an improvement. You could get a short-term contractor, but having someone on-board long-term who can weave testing into your whole development cycle would be much better.

I work in a small company and we have a written regression test that we go through for every release. It is not fun, but it catches bugs. We take turns and just do it.

Maybe you mean something different when you talk about end-users finding bugs. If so, you need to find a different way of expressing it, because that's a losing proposition if I've ever heard one. Specifically, it sounds like it loses customers. Advocate instead for:

• A professional tester
• A written regression test
• Test-driven development
• Establishing a culture of Quality.

Each of those is a time-honored industry-standard solution to a real issue you are experiencing that is affecting your bottom line. If you keep metrics, then you should be able to estimate a dollar figure to the number of users who leave because of bugs, and the negative advertising the bugs generate. There is also an opportunity cost, where customers who don't find bugs are happier and more likely to recommend your product. If 80% of the total cost of the bugs to the company is enough to hire a tester, that's your whole business case right there. A slam-dunk really. Good luck!

Answer 2

There are a lot of issues here.

I personally am completely against this as I think that applications should be tested by end users and not programmers as end users are much better testers than programmers.

This is exactly the wrong attitude. You programmers are the first line of defense against garbage leaking out. Your end users are the last ones you want doing your testing. For one thing, it's too late. Those bugs should have been detected and fixed long before the product went out the door. For another, your company is risking having those end users becoming ex-users when they finally get so sick of having a buggy product that they go elsewhere.

We don't do end user testing really at all. We don't have a QA team or anything of that nature. We don't have test cases for our projects that are fully laid out.

You may not need a QA team, but that doesn't mean you can forgo quality. All of you need to act like you are members of the QA team. You need to do end-to-end testing. Unit testing isn't enough. You need to take on the attitude that bugs shouldn't creep in. Nip them in the bud.

Ok, I am just a peon programmer at the bottom of the rung, but I am probably more tired of these issues than the managers complaining about them. So, I don't have the ability to tell them you are doing it all wrong.....I have tried gentle pushes in the correct direction.

There apparently is a problem with management here, but you are pushing exactly in the wrong direction. I suspect your management has a "get 'er done" kind of mindset is only paying lip service to quality. This needs to change. Your competition is going to steal your customers away from you with a better and cheaper product if it doesn't.

Answer 3

Manual testing by developers beyond 'yep, the app starts' etc., is a thorough waste of time. Automatic testing is worth doing, as is examining the causes of failures to see if you can reasonably prevent them happening again either in the same code or in other code.

It's ok to have end users test if you have nobody else. Ideally have that structured so that end users can choose to test or to use a more stable version, and make sure you have a good way of getting information back in the event of a failure.

End users might be the only people who could possibly think of, say, testing what happens when you're in a locale where converting "I" to lowercase doesn't give "i" (Turkish, and this bit me), or running the application from a DVD drive, or without any network connected. The trick is to involve them, actually respond to what they report and make them feel like they're contributing instead of being victimised.

Also, in certain environments you might pick a 'friendly' end user to test certain features/fixes. Especially if you can watch them use the program, you can learn a lot that way.

Answer 4

Developers should be doing real unit and integration testing that can be automated. Ideally after every check in/commit (or at least nightly) all of those tests should be run and pass before the code is released. There are a number of continuous integration tools out there can do this for you (e.g. http://jenkins-ci.org/).

When defining the tasks for your project, make sure your time estimates include developer time for building these tests. NOTE: Usually you will spend more time writing tests than you will writing the actual application code.

Whenever a bug is reported, start by adding a test that reproduces it. Then fix to the test. This will prevent the same bug from coming back.

As for changing management's outlook you can try the following: * Track every hour that is spent resolving bugs * Highlight the bugs that have come back repeatedly * Show them how much time is being wasted fixing bugs rather than improving and adding new features * If you have some downtime start setting up these processes yourself, so at least you know your code is good If you can show them in numbers the cost to the company it may change their outlook.

Answer 5

I personally am completely against this as I think that applications should be tested by end users and not programmers as end users are much better testers than programmers.

This is not a good practice. Your application functionality should be tested by programmer before handling it to the testing stage. QA should be responsible for testing of functionality as well as related modules in your application. After it past QA, your client users (not end-users) should have a UAT (user acceptance testing). Once it passes UAT then you deploy it to production.

However, if your programmers perform testing through writing unit tests, then this is is a great practice to follow.