Someone in my company recently proposed changes to our core product that our managers feel should trigger what I guess my company considers a full QA cycle (i.e. testing the entire product suite from the ground up). Apparently our QA takes 12 weeks to do a full QA cycle for our product. My problem with this is that we are trying to do Agile (although mostly half-assed in my opinion) development. We will do a whole set of sprints and then do a release, which QA will take forever to go through I guess. The question is really, if our QA is going to take 12 weeks to do their job, shouldn't we just give up trying to do Agile? What the hell is the point of trying to do Agile in a situation like this?
-
36I would venture to say that if QA takes 12 weeks, then you aren't "doing agile."– SingleNegationEliminationJul 28, 2011 at 4:34
-
10If team is not responsible for quality of code they produce, I would not call it agile, either..– merrypranksterJul 28, 2011 at 13:09
-
1@merryprankster Could you elaborate on your response? Do you mean to say it's pointless to not have QA be a part of the team and verify quality as part of the sprint? Or do you mean that the team should be verifying quality on their own to a point where QA should be rendered almost useless? Or perhaps another meaning? I don't know correct answers here. I'm just looking for any advice I can get on a way to rectify a situation that I feel is horribly broken. Thanks.– David HosierJul 28, 2011 at 16:05
-
2I mean that the team should own the quality process. They will do what it needs to be done to make sure that quality is good enough. This keeps feedback loop as short as possible, and it makes it more personal. Quality is not an external property, it's inherently part of development.– merrypranksterJul 28, 2011 at 16:20
-
2This becoming a chat, so this will be my last comment. Yes, I agree that in real world, you are limited by your environment. Also, you should be able to pick & choose your ways of working. However, I think it's not true chat agility is being flexible in every way, quite the contrary: agility requires discipline. One important aspect of agile development is keeping feedback loops short. If you have QA phase outside iteration, feedback is late. If team does not address QA as part of iteration, they are not agile. Team can decide how they do QA - that's flexible - but team should do it.– merrypranksterJul 28, 2011 at 17:48
|
Show 10 more comments
9 Answers
Well, the direct answer to your question would be Mu I'm afraid - there's just not enough details to make an informed guess whether you should or not quit trying.
The only thing I am pretty positive about is that level of agility should be driven by customer / market needs (which you gave no info about).
- For example, as a user of IDE I am perfectly happy to upgrade to new version once or maybe twice a year and I am never in a hurry to do that. Ie if their release cycle is 3 months (12 weeks) then I am perfectly happy with that.
On the other hand, I can easily imagine, say, financial trading company go bankrupt if it takes more than a month for their software to adapt to market changes - 12 weeks test cycle in this case would be a road to hell. Now - what are your product needs in this regard?
Another thing to consider is what level quality is required to serve your customer / market needs?
- Case in point - in a company I once worked we found we need some new feature in a product licensed from some software vendor. Without this feature we suffered rather strongly, so yes, we really wanted them to be agile and to deliver update within a month.
And yes, they appeared to be agile and yes they released that update in a month (if their QA cycle is 12 weeks then they likely just skipped it). And our feature worked perfectly well - guess we should have been perfectly happy? no! we discovered a showstopper regression bug in some functionality that worked just fine before - so we had to stick-n-suffer with older version.
Another month passed - they released another new version: our feature was there but same regression bug was there too: again, we didn't upgrade. And another month and another.
In the end we were able to upgrade only half year later so much for their agility.
Now, let's look a little closer into these 12 weeks you mention.
What options did you consider to shorten QA cycle? as you can see from above example, simply skipping it might not give you what you expect so you better be, well, agile and consider different ways to address it.
For example, did you consider ways to improve testability of your product?
Or, did you consider brute-force solution to just hire more QA? However simple it looks, in some cases this is indeed the way to go. I've seen the inexperienced management trying to fix product quality problems by blindly hiring more and more senior developers where just a pair of average professional testers would suffice. Pretty pathetic.
The last but not the least - I think one should be agile about very application of agile principles. I mean, if the project requirements aren't agile (stable or change slowly), then why bother? I once observed top management forcing Scrum in projects that were doing perfectly well without. What a waste it was. Not only there were no improvements in their delivery but worse, developers and testers all became unhappy.
update based on clarifications provided in comments
For me, one of the most important parts of Agile is having a shippable release at the end of each sprint. That implies several things. First, a level of testing must be done to ensure no showstopping bugs if you think you could release the build to a customer...
Shippable release I see. Hm. Hmmm. Consider adding a shot or two of Lean into your Agile cocktail. I mean, if this is not a customer/market need then this would mean only a waste of (testing) resources.
I for one see nothing criminal in treating Sprint-end-release as just some checkpoint that satisfies the team.
- dev: yeah that one looks good enough to pass to testers; QA: yeah that one looks good enough for the case if further shippable-testing is needed - stuff like that. Team (dev + QA) is satisfied, that's it.
...The most important point that you made was at the end of your response in terms of not applying agile if the requirements are not agile. I think this is spot on. When we started doing agile, we had it dialed in, and the circumstances made sense. But since then, things have changed dramatically, and we are clinging to the process where it may not make sense any longer.
You got it exactly right. Also from what you describe it looks like you got to the state (team/management maturity and customer relationship) allowing you to use regular iterative model development instead of Scrum. If so then you might be also interested to know that per my experience in cases like that regular iterative felt more productive than Scrum. Much more productive - there was simply so much less overhead, it was simply so much easier to focus on development (for QA to respectively focus on testing).
- I usually think of it in terms of Ferrari (as regular iterative) vs Landrover (as Scrum).
When driving on a highway (and your project seem to have reached that highway) Ferrari beats the hell out of Landrover.
It's the off-road where one needs jeep not sports car - I mean if your requirements are irregular and/or if the teamwork and management experience are not that good, you'll have to choose Scrum - simply because trying go regular will get you stuck - like Ferrari will stuck off-road.
Our full product is really made up of many smaller parts that can all be upgraded independently. I think our customers are very willing to upgrade those smaller components much more frequently. It seems to me that we should perhaps focus on releasing and QA'ing those smaller components at the end of sprints instead...
Above sounds like a good plan. I worked in such a project once. We shipped monthly releases with updates localized within small low-risk components and QA sign-off for these was as easy as it gets.
- One thing to keep in mind for this strategy is to have a testable verification that change is localized where expected. Even if this gets as far as to bit-by-bit file comparison for components that didn't change, go for it or you won't get it shipped. Thing is, it's QA who is responsible for release quality, not us developers.
It is tester's headache to make sure that unexpected changes didn't slip through - because frankly as a developer I've got enough other stuff to worry about that is more important to me. And because of that they (testers) really really need solid proof that things are under control with release they test-to-ship.
-
1I think this is probably the best response in light of our current situation. For me, one of the most important parts of Agile is having a shippable release at the end of each sprint. That implies several things. First, a level of testing must be done to ensure no showstopping bugs if you think you could release the build to a customer. Second, assuming the first statement is true, is it possible that QA is duplicating a lot of work that should have already been done during development? I think there is probably something to address there, both in our QA and our development team (I am a dev). Jul 28, 2011 at 16:32
-
However, I don't recall us ever releasing a build to a customer after a sprint. Furthermore, the way our customer base is, they don't want a full product release that often. Our customers are slow to upgrade. The most important point that you made was at the end of your response in terms of not applying agile if the requirements are not agile. I think this is spot on. When we started doing agile, we had it dialied in, and the circumstances made sense. But since then, things have changed dramatically, and we are clinging to the process where it may not make sense any longer. Jul 28, 2011 at 16:36
-
3Our full product is really made up of many smaller parts that can all be upgraded independently. I think our customers are very willing to upgrade those smaller components much more frequently. It seems to me that we should perhaps focus on releasing and QA'ing those smaller components at the end of sprints instead. We could shorten the feedback loop due to a more focused approach and deliver value to customers more quickly. Then we could do a full product release at some point that wraps up all the smaller bits. Then QA has less to do since most has already been validated in prior sprints. Jul 28, 2011 at 16:39
-
1+1 I like your examples of different market needs. One could provide more extreme examples. E.g. controller software to manage space rocket launches. Customer might be happy with upgrades every five years (the laws of physics don't change much), but just one single regression bug could cost hundreds of millions of dollars.– MarkJApr 26, 2012 at 12:04
Oh, I do feel your pain. There are some serious changes you need to make to the QA team for this to work.
My advice is to split the team into three teams:
Feature testing - Fast turn-around on testing new developments.
Regression testing - Fully testing the product before it goes out of the door. This shouldn't take 3 months, even after reducing the team size because most bugs will be found by the first team.
Automated testing - Writing a full suite of regression tests to speed up the job of the regression testing team.
The third team is a bonus, but if you can't have the first two teams then you may as well be waterfall.
-
-
I think this is a very good response. I'm not really aware of how the QA team is organized or how they proceed with their testing. Our QA team is in India, which I think is a not insignificant part of the problem. From what I understand, their test plans are not published such that anyone can review them and validate them. Furthermore, due to the time difference, the turn around time on the back-and-forth between developers and QA is atrocious. What should take an hour of conversation at someone's desk turns in to days of emails or JIRA comments. Jul 28, 2011 at 16:17
By way of illustration:
Note that your QA team is probably working outside the (ATDD) circle, and you are working inside.
I think it is OK to work that way; if you can prove in your automated tests that you are fulfilling the customer's requirements on each sprint, you can allow QA to perform their tests at their leisure, and come to you with defects, which you can then work into the next sprint.
answered Jul 28, 2011 at 5:39
-
3A problem is you are getting defect reports from work done 4-6 sprints ago (assuming 2-3 week sprints). Depending on the company's QA policies and procedures, they might actually have to sign off on a sprint before it can be released to the customer. So, yes, you have potentially shippable products after every sprint, but less than 25% of those will hit QA (assuming that when they finish testing one candidate, they begin testing the most recent candidate) so you can show a customer a product every few weeks, but they can only get one every 12 weeks and it'll be older than what they just saw.– Thomas Owens ♦Jul 28, 2011 at 9:51
-
Right. I was just discussing this with a colleague. I'd say we haven't even really been doing proper releases at the end of each sprint. We do a build at the end of each sprint because that's what Agile says you should do, but we have no intention of anyone ever seeing that build. I don't know if QA gets those builds or not, but I can assure that you no customer will ever see a build at the end of the sprint. Only one build is potentially official, and it's the one from the last sprint. In my mind, that's totally not Agile at all. With that workflow, Agile is only a way to organize work. Jul 28, 2011 at 16:10
-
Furthermore, I don't recall ever getting feedback from QA until after the build from the last sprint as I mentioned above. This validates your point. What I think this could lead to is situations where decisions that are made in sprint 1 turn out to be faulty decisions, but that faulty decision is not fully realized until all the subsequent work is done on top of that faulty decision. This of course could lead to a huge amount of rework. Jul 28, 2011 at 16:14
It sounds like you have a "Definition of Done" problem.
Given that your QA group is external, and only involved on customer releases, you can't rely on them for timely feedback on issues. That means if you want rapid feedback, you're going to have to bring some degree of testing "in-house" for the team.
Treat the QA Group as if they don't exist. Act is if your release at the end of the sprint will be deployed to your most critical environment the next day. The software isn't done until it's ready to go to the customers.
QA should find nothing.
This will be harder to get to. You'll probably have some things that sneak through the first few times. Automated acceptance tests and "regression" tests are your best friends here. TDD will help you build up large parts of such suites. You should be able to know -- quickly -- if you've broken anything.
Do you have a customer representative / product owner who can see a given release before QA is done with it and give you authoritative feddback on it? If so, you can do, and have most of the benefit of, agile methods while treating QA as a secondary, somewhat slow source of feedback. A release would be only "officially ready" after QA is through with it, but you wouldn't have to wait for them before starting the next.
But if the company rules say that the customer must not see a release before QA is done with it, then you can pretty much forget about being agile, until you manage to have those rules changed.
answered Jul 28, 2011 at 8:03
The process you described is not an agile process. Teams who have a high degree of agility are able to deliver reliable and potentially releasable builds every sprint. In most agile implementations, the QA function is built within the agile team helping to achieve this goal.
If you, your project lead, your product owner and the developers are not working together and you do not have an improvement plan (retrospectives) then name your process something else and move on. It does not appear that your teams problems are the fault of managers or QA. They seem to be reacting to some systemic problem coming out of the development organization. All is not lost if the team is willing to take responsibility and begin working with stakeholders.
You could try three things. One, make sure each stakeholder has concisely defined roles and that each person understands their responsibility. Two, stabilize the build and then get signoff from QA without introducing more changes. Three, institute test automation. The QA team will love you for it.
-
You are 100% correct. Your three items are good advice. I can only affect so much change as a single developer, but I can try to lead by example and see if some QA people want to come along for the ride. My biggest frustration is that nobody else really seems to care, which is obviously a huge barrier to the successful turn around required. Most people on the team are just happy to continue with the status quo; at least that's my impression. Oct 4, 2011 at 16:27
Its a pity the feedback takes so long but I don't think it is worth stopping with agile. At the end of a sprint (or a couple) you release a product you are confident it could be put in the market. For your team agile brings the ability to focus on the work to be done and keep the product releasable. When the QA finds issues I suggest to create bug reports for these issues and address them in the next sprint (if they have a high enough priority).
Our product field tests take a full 8 week plus that we are dependent of outside growers. Still by doing agile we are able to stay focused on the work at hand and produce a new version really quick when needed.
The problem lies (in your eyes) with the QA department can the problem be solved there? Have you discussed it?
-
Your response brought up some good conversation between a colleague and myself. The main point in your response that got me was, "At the end of a sprint (or a couple) you release a product you are confident it could be put in the market." I don't ever recall releasing product at the end of a sprint until after we've completed a whole series of sprints, it's gone through QA, and we've had several rounds of follow-up bug fixing. In that respect, I think we're using Agile merely as a way to break up and organize our workload and nothing else. We are not gaining any benefits of Agile really. Jul 28, 2011 at 16:22
12 weeks is long, but hopefully QA can provide you with feedback and bug reports during that time (rather then after the three months).
Then you can still respond to the most important issues in an agile way and can fix many if not all before they've even finished!
What are the QA people doing while you're executing multiple sprints? Sounds like they feel the need to test everything after every change (Which is why they wait for a whole bunch of changes.).
The development team is agile, but the rest of the company is not.
Whoever is in charge of QA either doesn't know what he/she is doing or they have performed a Jedi Mind Trick on upper management and are allowed to take their sweet time. How can QA take longer than development?
