There are already questions on SE that ask if a developer should have access to the production server(s).

But, given the fact that I already have a root access on the production server, should I say that, as a developer, using that access is beyond my responsibilities? Should I argue that I'm not a sysadmin and I am afraid to break things?

Or, should changing things on the production server be considered bad practice but just a part of our poor release/debug/development /whatever process?

Thank you.

  • Try at least to interact with the sysadmin person. – Basile Starynkevitch Sep 25 '14 at 5:37
  • Notice that part of the rule from "no developers in production" come to avoid developers doing dumb things like hardcoding URLs and other details that should be configurable. As long as you know WHY it is a good idea to keep developers out of production, you can both keep the root access and management and code carefully to avoid these issues. I am at the same position and still I always do DEV -> OSB from SVN -> DEV -> PRE -> PRO, because I know it is slow but I still know that change control is necessary. – SJuan76 Sep 25 '14 at 12:40
  • 1
    the rule shouldn't so much be "no access" but "don't change things without a paper trail"... If you have to analyse a production problem, having access to that server can help immensely. And you might need to do an emergency fix directly there to prevent multi-million dollar losses or claims, but FFS document those changes thoroughly and backport to every branch anywhere in the pipeline... – jwenting Sep 25 '14 at 19:25
  • 2
    I'm really glad I'm not the only person who doesn't want these kinds of responsibilities. – sevenseacat Sep 26 '14 at 3:42

It depends. Here is the meta principle. If you are in a small company, and there is a problem, and you're refusing to solve it, then your behavior just became a problem.

Now the question is what problem you have.

  1. Is the problem that production changes need to be made and you're the one available?
  2. Is the problem that you really don't know how to do the task?
  3. Or is the problem that someone is trying to use you as a back door to get things done that someone else thinks shouldn't happen?

Decide fast.

If it is the first, then what you should do is perform the task, carefully document as you go what you did, and then send that to whoever ideally should have done that job, possibly along with your comment that you feel out of your depth and this is risky. Feel free to CC that to any and all who you think are qualified to have an opinion on what you did. There is a chance that you'll get feedback that helps you do it better next time.

If it is the second, then measure twice, cut once. This is a learning opportunity, treat it like one. And send an email like the one described above looking for feedback and advice for how to do it next time.

If it is the last, then you should say, "So and so is responsible for this, and I don't want to create a problem because I don't know what else I'd be stepping on." And then stand firm on passing the buck.

If you are in doubt, err on the side of getting the immediate thing done after delivering any warnings you need. Usually you won't screw up (too badly), and that addresses an obvious immediate problem. If your action turned out to be wrong, blame tends to be easy to deflect. (Unless you're in a toxic organization, in which case the only choice that will work in the long run is getting a better job.)

| improve this answer | |
  • 2
    Good answer, but I think the OP should still be willing IN CERTAIN CASES to stand up and say "I won't do this because I don't know what I'm doing". If the production server is left in an unstable state, no one will applaud the OP and blame not be as easy to deflect – matt freake Sep 25 '14 at 8:40
  • @Disco3 This does come up. But that is where, "Measure twice" comes in. Try to figure it out, and only after getting blocked say, "I can't." – btilly Sep 25 '14 at 14:07
  • 2
    I tend to agree with @Disco3 - There's only so much measuring you can do on production. – Telastyn Sep 25 '14 at 19:06

Are you in any way shape or form Responsible for any action that must be run "as root" on the Production server?

If so, then you're lumbered.

  • Get hold of a shell add-in that records everything you type into a file - that way, at least you'll have a record of how things got broken,
  • Get someone else to look over your shoulder before you press anything circular and vaguely red (especially if its flashing to any degree),
  • Notify whoever looks after your corporate Security arrangements that you have this level of access (and let them ask why) - that may well stir up a hornets' nest for your Boss,
  • Investigate ways of minimising the risk of your causing any damage - can what you need to do be done any other way (or by somebody/ something else)?
    Automation - doing things reliably, the same way, every time, is a Wonderful thing.
  • Brace yourself for the worst when it happens.

If you don't actually need this level of access, then somebody needs to take it away from you.

The Principle of Least Privilege doesn't just apply to application code; it applies equally well to people. Does you Boss really want you in a position to see their salary? Letting that "slip" in conversation might very well get things rolling.

| improve this answer | |

Avoid interaction only if...

Completely avoiding the production server is only practical if:

  1. You provide the client with a painless way of updating the software, and you are willing to provide training to show them how to go about it.
  2. Updates are such that the client can perform them at their leisure. If you require them to perform an update for security loopholes in your software, they may not actually want to / be able to do so, and you must account for that.
  3. You reject the possibility of having access in any way. One of the legal implications of being able to access the production server is that you could be blamed for literally any type of problem that you have access to. Removing that possibility is paramount to removing legal responsibility.

However, the advantages of completely avoiding the production server only apply if the entire company is in agreement. If you, the programmer, decides this is the best approach and Bob, the intern programmer disagrees and ignores your heed, you've accomplished nothing ultimately except potentially some cigarette ashes in your morning coffee in the office. In other words, if this is not company policy, this is what must change, not you personally. You can mention these points to your boss, however, if your company isn't onboard, you must ultimately interact with the production server.

Some helpful advice if you must

Should you have to interact with the production server, I have enough experience to mention a few pointers:

  • DO perform backup before doing anything that would manipulate the database. The more recent, the better. Should something happen and you have a day-old backup, you'll still be in the sh*t, but at least you won't get fired over it. Have a copy from half an hour ago, and you may even get out of it unscathed.
  • DO provide notice prior to performing any activity on the production server. Should this be a legal issue, while this isn't rock solid evidence that you don't touch the server without a notice, it is a good argument in your favor. If you send an e-mail, be sure to get back an indicator that your customer has read it.
  • DO contact your client when would be convenient to perform updates / checks. You only really need to do this once, to get a general idea of hours (lunchtime or whatnot). Make sure you send a confirmation of the hours communicated by phone in writing in an e-mail, and make sure that e-mail is received by your client (e-mails do still fail sometimes).
  • DO NOT perform changes on the database that you cannot reverse. Yes, you have a backup, it's true, though how would your client react if you were forced to tell them that the only way to return to a previous state was to wipe out all data for that day? More importantly, how would your boss react to you telling him that you should wipe out the client's production data for that day? If there is no other way, you can always choose NOT to do it. You can always port the data to your local databases to replicate problems (assuming of course that you have approval). There should be no excuses here.
  • DO NOT leave junk data on the database. This is a corollary to the above statement. If you created a new record to test something, though it may not impact the functioning of the program, the client can still see this. It is completely unprofessional as a programmer and it makes your company seem like you sell chocolate, and also software on the side.

You probably already know this points or you wouldn't be asking if it is such a good idea to modify the production server, though maybe this answer could help you formulate some sort of company policy for what concerns interaction with the production server.

Hope that helps!

| improve this answer | |

The question of 'should' finds its way into a number of other situations. In the world of devops where programmers take what is seen as traditionally IT roles, the line gets very blurry. In other worlds such as banks or anything that handles money (or health care and hipaa), the question is a resounding 'no' with no further elaboration necessary.

Lets look at how one of those money handling areas handles it. The PCI DSS (Payment Card Industry (i.e.: Visa, Mastercard and the like) Data Security Standard) has some rather specific points on this subject.

There are specific requirements in the DSS such as 3.5.1:

Restrict access to cryptographic keys to the fewest number of custodians necessary.

or 6.4.2:

Separation of duties between development/test and production environments

This last one specifically speaks to what you are asking and in the guidance section reads:

Reducing the number of personnel with access to the production environment and cardholder data minimizes risk and helps ensure that access is limited to those individuals with a business need to know.

The intent of this requirement is to separate development and test functions from production functions. For example, a developer may use an administrator-level account with elevated privileges in the development environment, and have a separate account with user-level access to the production environment.

This means that if you are wearing a developer hat, you shouldn't have administrative access on the production environment. One can argue that you can switch hats (and all associated environments - here's the "you are on call and have admin access laptop that you don't do development on - and don't do admin things from your development machine."), but that gets into how management wants to claim they are meeting the requirements.

So, should devs have access to production? Sure - reading logs and the like is very helpful and a pain to have to go through IT and/or Operations organizations to get access to. Should devs have admin access to production? Depends on what they are doing and what the various regulations say.

If you are at a small shop that doesn't have the personnel, or infrastructure, or requirement that you don't have access you are likely to find yourself with admin access no mater if you like it or not and be asked to do admin things on it from time to time.

It really depends on where you are and how much red tape there is to protect production from the devs (because there are few things as fun as accidentally pushing code or editing live code on production).

The life of someone who has access to production is always 'interesting'.

I don't always test my code meme

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.