Virtualized Development Environments in Enterprise Networks

Question

We are attempting to implement a development environment using virtualization for a small team of 4 developers within an enterprise organization. This would allow us to set up separate development, testing, and staging environments - as well as allowing access to new operating systems that are requirements for systems or tools we are evaluating. We re-purposed an existing workstation-class machine, threw in 24GB RAM and RAID-10, and were doing fine until we attempted to get the machine added to the domain.

Now we are beginning the war that all enterprise developers since the beginning of time have had to fight - the fight for local control of a development and testing environment. The network and IT admins' have raised concerns ranging from "ESX Server is the enterprise standard" to "servers are not allowed on client VLANs" to "[fill-in-the-blank] is not a skill set currently possessed in the local or enterprise IT organization".

We could justify production-class hardware and formal IT support if we had to, but it would take time and involve a whole lot of headache. Even then it might take months to formally get IT resources assigned by treating this as a production system - and even if we did, we would likely lose the local control we need.

I imagine that many of you have had similar struggles over developer control of non-production environments - and virtualization in particular - so my questions are as follows:

1. What strategies and arguments have helped you win over the infrastructure (IT & Network) folks to allow these types of silos to exist within enterprises which have standard network and security policies in place that would generally (and understandably) preclude this type of non-(centrally)-managed infrastructure?
2. Have you found this to be a matter of technical justification - or more of a political struggle for control and ownership?
3. If you ended up with a IT-managed development environment, how much of a roadblock has it been for day-to-day development and testing?
4. Has anyone ended up moving their development environment to a disconnected VLAN or entirely separate network to avoid these network access struggles?

Also, this is not a Hyper-V vs. ESX holy war (we would be fine with either - but Hyper-V was selected since it is "free" with MSDN for these purposes [yes, VMWare has free tools too - but the good management tools generally aren't], and would be easier to manage by the local developers in a "Microsoft Shop") - so arguments for or against either are outside the scope of this question.

This is also less of a virtualization vs. physical hardware - I suppose the same question could be asked without the virtualization component to the equation.

Also assume that the dev team has already made assurances to either manage patch management and antivirus, or integrate with the existing enterprise systems if they will support it. This scenario, with different questions, is also posted on SF to hopefully elicit the opposing viewpoint.

Answer 1

You've gone "off the reservation", and are trying to justify it.

This isn't about virtualization; It's about control and responsibility. The IT department has responsibility for the safety and reliability of the company's systems. To make sure they work, IT keeps them under their own control. You've built a system not under the control of IT, and it's now becoming a problem.

The usual reasons programmers want their own systems, in my experience, is:

• IT is not responsive. It takes weeks to get a new environment, but you need one now.
• You need control; They won't give it to you. You need to be able to set permissions, install components, etc. IT won't let you.

Ultimately, when you go to production, you will want an IT-managed system that's completely locked down. But while you're developing, you need flexibility. Some suggestions:

• Make Friends. get to know some people in IT; Talk to them face to face. Explain your situation and ask them what can be done. You may be able to get admin rights to a dev server simply by asking.
• Run Local. If you can run portions of the application on your local machines, you may not need a server, or you can get away with a locked-down DB instance.
• Get a Sponsor. Nothing gets IT moving like a VP coming in and saying, "Why are you blocking my project?" Use your project sponsor's clout.
• To the Cloud! If your project budget will cover it, just host on EC2 -- you bypass your entire IT department. The risks are getting hacked, and getting fired for letting company information outside the firewall.
• Run the Long Game. Put in the requests for properly authorized and administered servers early. When you get complaints about your homebrew, say that you're still waiting on the official servers.
• Preallocate. Request servers that you think you might need in the future. Then re-purpose them when you have actual needs.

Answer 2

As much as I am an amateur in such situations as these, it seems that an appropriate and well constructed argument is required to justify to the heads of departments a need for the extra expense (and expanse) of the IT resources. You probably want a good speaker who is able to intermediate the issues and relate the potential value of the proposal to those who wind up paying for it.

The problem is actually one that deserves real consideration: One group want the Dev environment, but that puts some pressure on the other group who feel responsible, indeed are responsible for the security of the overall system, networking especially being something that IT depts are justifiably precious about.

It strikes me that the ability to off-site certain resources for a prospectively lucrative project or even just a free environment for developers has now been superceded by a market rationalisation for Virtualization as a cost cutting and resource control measure.

Now don't get me wrong, I'm not against virtualisation, far from it. But it occurs to me that there are often very good and explainable resons for permitting a development group to be entitled to a seperate realm which would be more productive an environment, and potentially safer than merely virtualizing everything.

Sure a business can save money by using the cloud for regular inter-office departental stuff, it's very useful there. (it's a form of virtualisation, but different, I know)

But suppose a developer raises an unidentifiable error that can't be debugged because there is a question about whether the application/program has broken due to an implementation of virtualisation (ie that it wouldn't occur in a stand-alone computer) then it becomes counter productive to waste time trying to track down the bug that isn't actually in the programming, but in the VM implementation.

I hope I am being clear. I don't have the answer for your specific case, but I think these are hopefully useful considerations in terms of the problem, and I would strongly recommend that such issues be discussed openly and fully with both departments involved, and perhaps a representative of the enterprise management who would ultimately have to make the case for purchasing. Hence my suggestion of a good speaker or intermediary!

Presumably if it requires more employees, then that might be a positive thing (there are plenty of unemployed people out there) but there might be enough IT smarts in the developer section to add a role like server admin for their own group themselves?

I know it's actually quite important, so I don't wish to be flippant, but there are times when I think the consolidation and addition of roles to existing workers lays far too much load on their personal time, which they tend to wind up resenting, especially when they could be being part of something radically new and successful software engineering.

I don't envy your problem, however I do envy the workplace that is fully engaged with bringing about new designs, new software, and new ideas. I sincerely wish you the best of luck, and hope my contributions are of some assistance.

Mihaly

Answer 3

The IT department actually have a point.

They probably manage thousands of applications on hundreds of systems. The only way they can do this effectively is to have a few selected standard software stacks running on even fewer standard hardware configurations.

If you follow this route you will encounter more and more problems as you get closer to production -- in the worst case scenario you will need to re-factor the whole application to run in a standard production environment days before you go live.

Better to work with the IT group and ask them to configure some standard test environments for you, its what they are paid to do. -- Ironically they will probably set up virtual machine for each environment.

Programers should program, let the IT infrastructure guys provide the infrastructure and the network guys configure the networks -- its how corporations work!

Also if your application is so non standard that IT will not consider building a test environment -- you will have zero chance of getting it in production. Talk to you enterprise architects find out what environments are standard and try to use them. If you really cannot implement your application using the standard software/hardware you need to make a formal request for enterprise architecture to approve your infrastructure as an exceptional case.

Answer 4

You will have to make your case with management that:

1. Having the virtualized environment fulfills one or more of the company's specifically stated requirements (such as the flexibility of supporting multiple platforms), and

2. You can implement it in a more timely manner, with less cost than IT can, and

3. Having local control will lower costs and reduce time-to-market delays, and

4. You can satisfy IT's security and maintenance concerns, and

5. Programmer productivity will not be affected.

The last one is a big if.   I have discussed this issue with a number of people that specialize in this kind of virtualization. They tell me that, by the time you throw enough hardware at this to make it as responsive as a local PC, there won't be any hardware cost savings.

So your demonstrated cost savings will have to come in the form of flexibility in configuration, and the ability to change those configurations at a moment's notice.