I work on a large software program in financial services. (15 project managers, 20 technical leads, 15 environments, 150 people on the technical side).
We do lots of Bankwide integration to hundreds of systems (insurance, bill payment, mutual funds, insurance, tax reporting, equities trading etc), which are frequently down in the development environments. Whilst there is an environment team, they're basically system administrators who need the assistance of a Java Lead to identify the root cause of an issue and fix it.
In smaller scale teams I'd worked on before (an investment fund system) a single PM would own a set of environments all the way to production, and would be responsible for removing blockages for a particular feature all the way to production.
In this larger programme, the project managers have a pattern of wiggling out of this responsibility. There are no points for them in fixing environments. In addition, tech leads get slammed for working on something that is not shipping features.
The test managers throw up their hands because the testers can't login to a system 1/3 of the time.
Now the way I've phrased the question may lead you to the answer "well change the way people are measured! Duh!" If you can articulate a concrete way for people to be measured that creates this incentive, I'm keen to hear it. Unfortunately things are not that simple. Project Managers commit to shipping software, and a kanban board shows everyone's utilization shipping stories, and so there is an incentive to maximise utilization and story points shipped.
Now you can make use of the information radiator and show all the stories as blocked, but the answer that comes back in that situation is, "make it someone else's problem," instead of "take the time to find the solution, and fix it in a way that it never happens again."
Another arguments is that this sort of thing sorts itself out. The person who feels the pain needs to spend the time to fix the problem. Interestingly enough this doesn't stop them from being slapped by the PM for letting their utilisation go down.
I'm considering taking it to the top of the program, and offering a simple 'washing up roster system' - that puts one PM/Tech Lead pair on fixing the environments a day per fortnight. The feedback I've had on this is that the Programme head finds it convenient to ignore these issues, or fire off a short-term operational responsibility "you - make this go away!" instead of thinking strategically about the problem. Taking it to the top is basically playing with fire.
When I take it to the test manager to ask him to talk to the head of the program, he says "The head of the programme is an operational not a strategic thinker. He's not interested in a systemic or medium term fix."
My current idea is to get agreement from the test manager on the burn-rate costs associated with environments that are down - and then link these to our automated availability reports. (We have reports that show graphs of different parts of the system being up and down). This way we can have an argument about the cost of fixing vs not fixing. The problem is that this is tempting a reaction because it relies on making people look bad financially.
My question is: What are the strategies on a large software program (200 engineers) to get people to fix environments when people are measured on features?
EDIT: Thanks the feedback so far has been enormously constructive and helpful. The question was raised about what an environment issue is. These include but are not limited to:
- The primary integration and customer web server is out of memory
- The primary integration web server hasn't loaded its caches
- The primary integration web server has failed startup and is not showing a login page
- The primary integration web server is out of system with the Tivoli access management system
- One of the multiple of satellite systems is down (emails, statements, fees, equities trading, user setup, end of day)
- The primary database being down or running slow
The broader point being that the rate of change on the system is high enough for this to be more likely new issues rather than the same issues cropping up.
Someone helpful has suggested systematising these and measuring the occurrence of them. I have proceeded on several 'lightweight runbook' initiatives listing issues and root causes on a wiki. The more poisonous PMs see this as a utilisation failure.
Someone helpful asked about the definition of done. At present this is defined as the software passing a DEV/QA test. (Ie prior to the SIT test phase, the UAT test phase, and the performance test phase).