To always have a potentially shippable product at the end of each 2-week Sprint, we need to rerun a lot of regression tests at the end of each Sprint. This is very repetitive and time consuming, so eventually it will probably be cost-effective to automate these tests.

However if we don't require a potentially shippable product, but plan for a few stabilization/hardening Sprints, we don't need to rerun all our manual tests each Sprint. In such case automating our tests will not be cost-effective.

So which approach do you prefer, to automate tests, to forgo the shippability advantage, or do you think that this trade-off doesn't actually exist?

closed as primarily opinion-based by gnat, Sklivvz Feb 14 '14 at 16:06

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Do you really care what anyone prefers or would you rather hear reasons for one or the other so you can decide for yourself? – JeffO Feb 14 '14 at 14:19
  • Both, honestly. – Eugene Feb 14 '14 at 14:27

At the end of a sprint, whatever you deliver should be of production quality, that's one of the pillars of scrum. If it's not done at the end of the sprint, it could drag on for ages as you find new stuff to add and new stuff to fix. Get it done and get it out the door.
If you need a reason: because you want that feedback only customer usage can get you.

If I dissect your post, what I'm reading is that you're having problems with bugs either way. It's not by running the tests that those bugs will fix themselves, so you're going to wind up with "hardening" sprints regardless.

My advice:
1)Invest in building quality in. If you're having problems with bugs, get the team together and find out why. Don't lay blame, just pose the problem and search for solutions.
2)Are your tests actually in code? Because coded tests are usually trivial to automate.
If your idea of a regression tests is going through some screens manually, you're going to have to brush up on different levels of testing. UI tests can be automated, but they are the most difficult, so you should try to test as much as you can deeper towards the unit level.


I think you need to approach this solution in a sprint fashion.

You want to compare the time spent creating automated tests to running them manually. We'll assume you know the cost of doing your manual regression tests or this whole exercise is moot. The key is how effective your team is at creating and maintaining automated regression tests. If you have no experience in this area, this project may not be the time to start learning because you're not going to be very efficient and probably are unable to estimate the amount of time they will take. Still, you can try automating one or two of your tests and see how much time savings you can get along with being able to estimate the time to automate the rest (automation doesn't have to be all or none. Some tests may be easier to just do manually).

Do you know how many sprints are left? Very difficult to do an ROI on comparing savings on two approaches to a repeated task when you don't know how many times you'll repeat it. Maybe you are fairly confident you can come close, but if you feel there may be more sprints involved in this project than you think, even the slightest amount of time savings with automated tests may be a morale booster. If team members find themselves hating doing the tests and would rather tweak pixels in the GUI, automation may save your sanity.

History of projects with and without any integration testing. Did your team choose to do testing because they dreaded the projects that didn't have it or are you just following the fashion trends? This can be another sanity check. It also is dependent on the number of sprints left. Having a lot of sprints left may not be a sign of "just think how much time we'd save by not doing any regression tests" but "there are a lot of sprints left because this is a very complicated project, so having tests may be more beneficial than we think".

I don't think there is a universal answer for any team to test or not to test and how much should you automate them. Experience has a lot to do with it. Some projects are more conducive to implementing new things more than others. If the though of automating tests is going to throw your team in a panic that this will put the project too far behind, don't do it. On the other hand if they feel like they'd rather spend their weekend writing automated tests if it means they no longer have to keep doing it manually (Isn't this why we wanted to learn how to program?), that may be the route to go. What's the price of a happy team?


I work in both styles simultaneously, with legacy code thoroughly regression tested only just before an actual release, and more current code tested with automation.

If I had the ability to make a choice to work in only one style, there is no contest - automate the tests and run them as often as feasible. The code quality is better and issues are discovered while developers still remember what they were thinking about. They're a lot quicker and easier to fix that way.

Not the answer you're looking for? Browse other questions tagged or ask your own question.