Understanding the problem when things break in production

Question

Scenario:

• You push to production
• The push broke multiple things
• That same build did not break qa or dev
• As a developer, you don't have prod access.
• There is lots of pressure from above to get things working agian.

Specifics:

• PHP/MVC application that is API-driven in Zend.
• Deployed to a few servers.

My question:

While investigating, lets say I have a hunch that something is wrong. But, I don't know for sure. And, of course, I can't test things in production. If I have a suggested fix based on that hunch, would it be wise to try and apply it and see if it works, before understanding what the problem is?

Answer 1

Grab as much information about the problem as you can (logfiles etc.) and then rollback the production servers to a working state. That's a pain from the developer's point of view of course, but is most likely a given.

Next, try and see if you can reproduce the problem in a development environment. If you can, then fix it and try releasing again.

If you can't reproduce it, then see if you can add more diagnostics and release to one server for a short time to get more information about the problem.

If that's not possible then look more closely at the differences between production and the dev/qa environments and try to make a dev environment closer to production.

Answer 2

How well do you understand the problem? What's the risk that your hunch will make things worse? Is it possible to go back and reproduce the problem in DEV/QA regions? What can you do to sync your DEV/QA region to get it closer to PROD? Maybe you have to change some environmental or database settings, maybe you have to import the PROD data to DEV, maybe you have to change some debug setting.

In general, I would not recommend pushing your hunch of a solution to PROD unless you can confirm that it is indeed correct in another region. I understand the sort of problems that come up when a bug happens in PROD and can't be reproduced anywhere else. That's when it comes down to seeing what else differs between DEV/QA and PROD and focusing on those. In my experience, it's usually an environmental setting or some configuration that is different, specifically for PROD. And I know that there is probably a lot of pressure from above to fix this, so is it possible to roll back to the previous working state and then try to reproduce the problem in DEV, come up with a fix in DEV, and then try again in PROD? That's what I'd suggest.

Answer 3

Depends on the kind of fix. More often than not, problems in production that don't appear in dev are related to contend in the database. So applying a bug that changes database content without being sure what exactly is "out there" may be a first step in a big disaster. If you can easily take the change back, you may be able to try around. But in general, if you don't have direct access, there should at least be a copy of the database or the whole server for tests. People with the right privileges would still have to run the new code, but at least without risk of data loss. (But sometimes size of the database or complexity of infrastructure prohibits such a setup)

It's really difficult, since there are many possibilities like different settings, libraries and versions of software.

Maybe you can write a piece of code first that evaluates with some debug output if your guess for the source of the bug was right and only then apply the actual bugfix.

Answer 4

Usually it is either configuration or data issues, assuming that the code and DB are identical between Prod, QA, and dev.

I would first look at the following:

• Any logging data your code has.
• Check the event viewer for unhandled exceptions.
• Check the data representing the progress of your application, it can be in the DB, files, etc. Does it make sense or not? Is what you expect?

Once you understand what is going on, you need to rollback the production to a working state and work on fixing the problem in a lower environment, till fixed and re-deployed to production.

Answer 5

While your environment is PHP, I have done a presentation on how to think about it for Java: http://www.infoq.com/presentations/maintaining-production-java-apps

The core issues are the same - to understand possible choke points to troubleshoot the situation: network, filesystem access, log files, deadlocks, etc. Also, to know how to ask the right questions: "System down" - "What specifically do you mean: Is web page slow, is there specific error message, is there timeout", etc.

Plus there are some tools to make troubleshooting easier: Wireshark for network troubleshooting is an absolute best and is worth learning. Others depend on the O/S you use. For Windows, anything from SysInternal (now part of Microsoft) is brilliant. For Unix/Linux, look at truss/strace.

On access to production, operations group should either know how to use those tools/techniques or you have a business case for them (together with you) to learn how to use them. After that, they need a specific set of troubleshooting protocols to run when problem happens, so you could do your analysis offline.

Answer 6

Short answer: Not if you have a choice.

Long answer: If you don't understand the problem, there are several risks involved in such a patch:

1. You could break something else, which could even be less reproducible.
2. You could just mask the problem, making it harder to notice and reproduce (which makes it worse)
3. You're throwing potential domestic experience away - experience which could make you a better programmer, and at the same time more valuable to your company (i.e. a potential future raise).

On the other hand, I see no harm in first checking if your hypothesized fix works, and if it does - then dig deeper and find out the real reason or other possibly better ways to solve the problem.