How do you QA and release software quickly with a large team?

Question

My work used to be a smaller team. We had less than 13 devs for a while. We are now growing rapidly, and are over 20 with plans to be over 30 in a few months.

Our process for QA'ing and releasing each build is no longer working. We currently have everyone develop the new code, and stick it onto a staging environment. A few days before our weekly release, we would freeze the staging environment and QA everything. By our normal release time, everything was usually deemed acceptable and pushed out the door to the main site.

We reached a point where our code got too big so we could no longer regress the entire site each week in QA. We were ok with that, we just made a list of everything important and only covered that and the new stuff. Now we are reaching a point where all the new stuff each week is becoming too big and too unstable. Our staging environment is really buggy week after week, and we are usually 1-2 hours behind the normal release time.

As the team is growing further, we are going to drown with this same process. We are re-evaluating everything, and I personally am looking for suggestions / success stories. Many companies have been where before and progressed beyond, we need to do the same

Answer 1

You need to set up a continuous build and automate your testing, so that all your unit tests and integration tests are run continuously (ideally at each repository revision if possible). If you set up that kind of automation, it is fairly easy to say that the program most recently passes QA at revision XXX, and then you no longer have to do a separate QA, you just need to sync to that version and release.

Answer 2

Ramping up too fast makes everything SLOWER. Has anyone on your team read "The Mythical Man Month" by Brooks? As each new person comes on board someone has to spend time getting them up to speed. And every decision takes longer.

You might try agile if the powers that be will allow it. Good luck.

Answer 3

If you don't have unit tests you need them. They will prevent a lot of bugs in the normal flow of you program. You also need to have a daily build that is QA by someone. Who reports the defects early. If you are fixing bugs as you develop you get better results then just fixing bugs at the end of the sprint.

Answer 4

Yes as other said, continuous build, automated tests are a good thing, read The Mythical Man Month

Maybe you should also try "one branch by feature and only merge once it is tested" and splitting in small teams.

But one thing you should do is stop growing until you are ready for it. Event if your pointy haired boss thinks that a blind squirrel is more likely to find a nut if there are a lot of blind squirrels. If the situation is complicated at 20 people, one thing is sure : adding more people will just make it worse.

And... when you are ready for it, grow slowly.

Answer 5

I would suggest that you reorganize:

Have a Q&A responsible officer which needs to review ALL code going in the release branch, and who is the only one who can merge in said code. Same officer may reject code if it is buggy or not up to coding standards. This person will most likely not have time for coding.

You will most likely also appreciate having a source control system that is powerful enough to facilitate this line of work, with multiple people working independently on the same project. I know that git can (because that is what Linus use it for), and hg and bzr may be able to too.

Halt development NOW, get a stable release pulled together, and initiate this workflow, and then slowly get up to speed again. You probably have too much inertia to be able to do this in-flight.

Answer 6

You automate the testing. Every bit of it. The QA team should be writing automated tests, not writing and executing manual test procedures. Ideally they should be writing the tests before the code is started.

Repetitious manual testing is simply unacceptable. Expensive testing is the root cause of almost every other problem that afflicts software projects.

Answer 7

The staging environment should never be buggy. You need to get a relatively continuous build environment working. I have had good success with hourly builds. Automatically, building and testing the staging environment daily should help. Automate your testing, and if if possible coding standards. Make sure developers are notified when they break the build.

Most revision control systems can run pre-commit checks which will can do things like verifying code follows standards. (Production will slow down as developers need may need to bring modules up to standard as they work on it.)

Assign some developers to review the changes as they occur. Assign review ownership of various modules to particular developers. (They should not be the ones changing the code they review.)

Branch on release point. This may require a separate staging environment if you don't want your staging environment bouncing back and forth from development to release streams. Minimal changes should be required in the release stream. Those changes will need to replicated appropriately in the development stream. Some changes can be merged back, but others (quick-fixes) will require rework on the development side.

Answer 8

The more people on your team, the more problems you are going to face--many times from the sheer amount of communication needed. When the team was small, the small mistakes that flew under the radar didn't affect as much as when you have at least twice the number of people--because the mistakes compound.

It sounds like you need to focus on your QA team to get them more agile and deal with the sheer size of the application.

• Focus more on automated testing--and train your QA staff to set that up. All critical acceptance criteria should have an automated test
• Have the QA team set up and monitor your continuous integration servers
• Find out where things are breaking down.

Agile works best with small, focused teams. You may need to break up the application into subsystems, and assign the sized team you had the most success with to that subsystem. Of course, this means that integration testing is all the more important. If you have a tiered QA structure where the small teams have at least one or two QA people to make sure that system is doing what it needs--you'll need another QA group that makes sure that all the software system as a whole is still doing what it needs.

Answer 9

I currently work on a ~30 person development team that frequently releases a very large codebase of significant complexity, so some of our experiences may be of interest.

When it came time to setup our test approach, we initially split off a team of 4 people to setup the "harness", the core structure of the code supporting the automation of our tests. All tests are written in Cucumber, and the step definitions are relatively light, mostly making a couple calls into the harness that does the real work.

Once the harness was to the point that the team was able to write tests, about half of our entire group transition over to writing tests. The exact balance onbviously depends on the level of testing needed for your own project. Some things we learned early on:

1. Create a couple tests and then loop back and harmonize them. Start building a standard so that new tests match the style of what is already in place. We ended up doing this later on, and had to go back and standardize what had been written so far.
2. Code review feature files. Aggresively look for step definition duplication, when people go head down writing these things they start missing that someone already wrote the exact same thing.
3. Make sure you are testing the right thing. For example, if submitting a form should do X, make sure the test is about submitting the form and not about clicking "Submit". The step definition may click the button, but the feature file shouldn't have to change if the name of a button does.
4. If a test is likely to change because of a change in implementation without a change in functionality, then the test probably will break down the road. This mostly goes with 3.

We got to the point that running all of our tests from start to finish took an obscene amount of time. If you think you may get to the same point, think early about automating test environment setup so that you can easily run tests across multiple machines. We can deploy our entire production system to a clean host in under a minute, which allows us to scale up testing in burst very easily.

Whenever we prepare to do a release we cut a build, and then run it through these tests. We review any failures, and determine if they need to be fixed before delivery. If they do, fix and repeat. The entire process from first build to delivery usually takes under a week.

We have just recently gotten to the point that we are integrating together our Dev and QA teams. Now, any feature we are going to develop has a feature file written before anything else. Then we develop the code for the feature. And right after that the feature file is run. This only becomes possible once you have enough people on your team that understand BDD/your harness that they can help the rest of the team produce good tests.