Developing on a production server

Question

Today I got yelled at for developing an application on a production server. Quote, "developing on a production server is not acceptable - ever!"

Here is the situation.

1. I set up a development instance: http://example.com:3000
2. The production instance is: http://example.com
3. I complete all my development work on http://example.com:3000 and when the client is pleased with the changes, I move them over to http://example.com.

The application I am working with is an old Ruby on Rails application, and I have to say that initially, I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

Again, am I an idiot? Probably so, but I've been doing web development for a couple of years now, and I have never encountered a situation like this. Who is right and why?

Answer 1

I used to develop on the production server. It can work fine, but it is inadvisable for at least two reasons:

1. Development code can cause infinite loops, memory leaks, or other problems that lock up the CPU, eat up all the memory, or otherwise affect the server in a way that will impact your production code.
2. If you need to make changes to components of the server environment as part of your development work, like the version of Ruby or MySQL or whatever, you'll be in a bind.

Answer 2

As others have stated, coding on the PROD environment exposes your users to your bugs. Even if you've started a different instance, you've still got shared hardware resources and can still access production files and databases. And as some of the comments point out, if your Dev instance gets hacked (for example, because you forget to wipe it and someone then discovers a massive security exploit in Rails), you've now got a publicly accessible machine with your app acting as a gateway in. Which would be... unfortunate.

Different businesses have different responses to this, but it can be generally broken down like this:

• Did a screw-up occur?
• How long would it take to revert a change (I primarily work in C++, so rolling back a binary can take significantly longer than in Ruby, especially when you've "lost" the old binary and have to recompile)
• What the effect of the change (rough guide: screwing up stored data is so much worse than not storing or displaying data, which in turn is worse than not showing the page at all)
• If you screwed up then walked out the door, would anyone know what you'd done?
• Was there another deployment option that would have prevented/minimized/detected the screw-up before impact?

This gives you the final calculation:

• How much would this completely preventable screw-up have cost the business?

This is now how much less your entire management structure is worth to the guy making budget decisions. Hence shouty.

If you're working on the company's internal "About Us" page and typo your own name to be L. "God-like" Thomas, embarrassing nickname problem; if you're working on the business-critical purchasing app, and it ends up accidentally plain-text debugging out credit card details to the homepage... lawsuit problem. Between those extremes lie everything from mischarging, crippling productivity, and all the other things that can drive away customers.

The reason not to allow it even for the "About Us" page is because coding directly in production is addictive. You start by only doing it for the minors, but over time, it's so much quicker not to have to get the DEV env up to scratch.

Beyond that, the size of the business can have a big effect. In a two-man team, when something goes phut, you lean over your shoulder and go "Oi, jackass, put it back". In a 300-person company, you have to start worrying about whether it was incompetence or malice, managers can be held responsible for stuff they had no control over, etc.

At the end of the day, if you follow procedure and screw up, they check what's wrong with the procedure. If you skirt procedure and screw-up, it's now your responsibility alone, even if the blame gets spread out a bit. Whether you want to roll the dice on that is up to you.

Answer 3

I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

I DO support the statements to AVOID development on a production server. You may only be justified to do under the GUN, if it is a typo correction in config file and insisted by your manager.

WHY NOT? - Because, it is very risky and pron to become a habit later on that would catch you up badly. Because, fatal production errors/failures may cost you to be fired from your work.

Let me re-iterate it again, even-though if you requested to do typo correction on production server, first do it on Staging. or in another words test your changes, test it, and again test it before placing in production.

This is something that happens often in places where culture of "do it quick and dirty" seems to be a norm.

Edit: Developing on production server, as a separate environment is also NOT acceptable. Any issues caused in your work may simply bring down the production server, and affect the production application performance. As example, i do remember a case when there was a security hole, when my previous co-worker tried to use production server WinServer 2003 for development purposes.

Answer 4

This is really a protocol issue. Generally, this is not something that you would want to be doing. You want to leave production machines alone. They may contain sensitive data, and you don't want to compromise the performance or stability of production sites with non-production ready code.

That said, there are times where this is commonly done. If you are in a position where you are pumping out low traffic brouchureware or simple CMS sites, this will likely be less of an issue. But if you are working on anything with sensitive data or "business critical" processes, you really shouldn't be risking having development code on the same machine.

Answer 5

Another important reason not to develop directly in production is that a development instance will usually produce and show verbose errors and stack traces. You never want to expose that to the user.

Yes, you can log them instead of showing them to the client, but that makes debugging that much less amusing than it already is.

Added Addressing your side-problem of having trouble with your development instance: I've had great success deploying a VirtualBox-based VM which duplicates our production environment (exclusive of hardware, of course) with an Ubuntu Server.

Answer 6

I'm quite astonished no one mentioned the most important reason yet, why it's absolutely forbidden to develop on production servers:

Don't mess with production data, which can happen oh so easily!

A tiny error in one place leads to gigantic troubles in other calculations and then, on the next day, all the data is garbage and the customer is pissed. This is much, much worse than a bug in the UI or a little crash here and there.

For most applications, the value lies in the data and not in the routines.

Answer 7

I always try and ask other developers what the procedures are for the particular company. In general yes, you should always:

1. Build locally.
2. Push it to some type of box that mimics production as much as possible to see if it plays nice there.
3. Possibly push it to a QA or certification instance to pass to the client/QA team to review the changes.
4. Push to production.

You can use Capistrano recipes paired with GitHub to handle all of this stuff for you. If you have to do this by hand every time, it can get old fast.

Answer 8

Another problem with developing on prod is that, sometimes, these things get missed in source control and a future release might undo your quick-fix change.

If you are in a publicly traded company in the US, you should not even have access to prod for regulatory reasons. In general, in no company should a developer have prod access (for the resons stated in all the answers as well as possible legal reasons), so your manager is also in the wrong for allowing you the rights to that server.

Answer 9

Rules that use "always" or "never" are usually ill defined. There will be edge cases where breaking a best practice will be justified. Much better advice will be "Don't touch the production servers unless you have very good reasons"

In my career I have found only two reasons to change code on production servers:

1. Bugs or behavior that happen only there and are not reproducible on the development environment. These are rare but can be very annoying and hard to find.

2. Directly fixing critical bug that you just cannot afford to wait to pass trough the normal deployment process if it takes more than a few minutes. After this has been cleared with management. If you are lucky you should only have few of those for your entire career.

Both are best left to senior developers that know the systems intimately.