Recently I made a program. I forget to delete 2 line of codes. That mistake cost me $800 per day every day.

I was programming with PHP. If a visitor uses proxy it redirect somewhere else. Using debugger was impossible because some code contains ioncube. Because the program simply redirect somewhere else no matter what, it's hard to see which part of the code is executed.

So I put a bunch debugging info everywhere. I thought I'll delete them latter anyway.

The most natural way to debug is of course to put debugging info into a file. The problem is I often use proxy. So after I change the program, I often have to download the text file with filezilla. Often the text file does not show what I think it should show. Finally I decided to just display error on the web.

I considered having debugging mode. However, I am afraid I will forget to delete debugging info.

I considered having debugging mode if user do ?debuggingmode=1 for example. However, I was paranoid that somehow my competitor can guess the secret keyword.

I deleted most debugging info. I forget to delete one and that one only show up if users use proxy from the right country. Turns out I do not have proxy from the right country and didn't realize that. After the program works for 24 hours, I uploaded that to my main domain.

My competitor, using proxy, see the debugging code. He copy the idea and that's how I lost $800 per day.

In retrospect, I really have a hard time seeing where I went wrong. I have been super careful. Yet it happened.

How should one debug a PHP web application securely without exposing secrets to competitors?

  • 5
    There's no such thing as being absolutely sure about anything, let alone bugfree software.
    – Tar
    Jan 13, 2014 at 10:27
  • 1
    Thoroughly testing again and again after each change made to the program/application even if it is very small change.
    – Rolen Koh
    Jan 13, 2014 at 10:51
  • 16
    "How should one debug a web application securely without exposing secrets to competitors?" - by creating a test environment that mimics your production environment. Live debugging should really very rarely be necessary.
    – CodeCaster
    Jan 13, 2014 at 13:08
  • 8
    I wonder what can be so critical about two lines of debugging code that it is worth $800 per day. Does it dump your private crpytographic key?
    – Philipp
    Jan 13, 2014 at 14:19
  • 2
    If you were earning over $800 a day for a while before this... does it even matter? But yeah don't put debug code on live! You could have a config debug mode boolean in a file. Have debug code only print if debug == true. This is a quick and dirty way at least, not worthy of being an answer! Jan 13, 2014 at 16:22

8 Answers 8


I really have a hard time seeing where I went wrong

The major mistake was that you reinvented the wheel. Instead of using default mecanisms for logging, you invented your own, which displayed the information within the page. A logging framework would rather store logs in log files, letting you to consult those logs later by SSHing to the server.

As for the bugs, producing bug-free code implies using specific techniques such as formal proof. Given their expensiveness, those techniques are appropriate for life-critical applications such as applications which control aircraft traffic or space shuttles, but are an overkill for nearly every business application.

■ See They write the right stuff in Fast Company magazine.
The article describes the methodology used at NASA, as well as the cost of producing software this way.

■ See Mechanizing Proof (Mackenzie 2004).
The book summarizes the history of automated proof of software, explaining the pros and cons of such proof, as well as the reasons it's not commonly used by businesses to write reliable software.

This being said, there are a bunch of techniques used for business applications to ensure software quality. Those includes but are not limited to:

  • Informal code reviews,
  • Formal code inspections,
  • Testing,
  • Personal desk-checking of code,
  • etc.

■ See Code complete (McConnell 2004), Programming Productivity (Jones 1986a), Software Defect-Removal Efficiency (Jones 1996), and What We Have Learned About Fighting Defects (Shull et al. 2002).

Also, don't forget continuous integration and continuous delivery. It helps in automatically rolling back the app in production to a working version when a revised one appears to have an issue which was missed during code reviews and unit testing, but caught once the app is deployed.

■ See The Secret to Safe Continuous Deployment (video)
It explains what techniques were set up at Google to prevent bugs which couldn't be found before deployment from staying for too long in production. It also describes pdiff and how was it used to catch bugs, including ones which were unrelated to presentation layer.


You should never debug in production.

You should always have a test environment which is identical to the production environment and debug there.

When you need to change code in the test environment (like for adding debugging statements), you should make sure that they don't go into production.

A professional setup usually looks like this:


The "Production", "Staging" and "Development" instances of your application should be as identical as possible so that a bug which occurs in "Production" can be reproduced in "Staging" and "Development", but still be completely separated from each other so that whatever happens in one of the instances doesn't affect the others.

When you need to analyze a problem, you do so in "Development". Mess around with debug statements and experiment all you want. When you found a solution, you apply that fix to the unchanged codebase in "Staging" and verify that the fix works. Then you promote the fix to "Production".

A proper version control and build management system can help you with that.

  • 1
    That's what I did. I debug in other domains first. After that, I move to the new ones. However, the bug on that other domain was still there.
    – user114310
    Jan 13, 2014 at 14:38
  • 1
    @user114310, Your development/testing environment should simply not be accessible by the outside world (either being on localhost or requiring VPN, or similar). If you inserted some debug code and didn't remove it before pushing to production, that's human error and there's nothing technological that can protect against it; simply be more careful.
    – Brian S
    Jan 13, 2014 at 15:26
  • 3
    @user114310 Sorry, I don't understand. You fixed the bug in staging but when you applied that fix to production the bug was still there? That would mean that you didn't reproduce the bug correctly and accidently fixed something else. When you are unable to reproduce a bug in development, it means that your development environment isn't identical to the production environment. When you discover that you need to fix that first before you can research the bug properly.
    – Philipp
    Jan 13, 2014 at 15:29

This is rarely done because the effort is not worthwhile. Even if you lose $800 a day, the effort of proving a program correct quickly becomes larger than that, which implies that there is no business case for doing it.

If being certain is worth that much (e.g. for Space Shuttle software or missile control), then you perform formal verification, exhaustive testing of all possible inputs, etc. Granted, it's also extremely difficult as well as slow and expensive. But projects with billion-dollar budgets also tend to have extremely bright people on them. (Or maybe they just used to - present-day headlines seem to contradict that rule.)

  • 1
    Even NASA gets it wrong. You can put as much effort as you like in, but some things are simply beyond human comprehension and thus very very difficult to assure.
    – Phoshi
    Jan 13, 2014 at 10:39
  • 1
    +1: Formal Verification is a thing, It would be good if you can go into more detail about it. I know it is is a thing but I don't have the words to find more detail. Eg verification by testing all inputs, reduction to finite statemachine, reduction to first order logic. What is the word for this? Is this Verification by proof? I think it is. Jan 13, 2014 at 15:23
  • There is formal verification but there is also heuristic verification that, while not certain, can provide verification to a certain confidence level. I don't remember much on the topic from college but one tool we used in the course was Spin which I believe is capable of exhaustive verification as well. Some of the descriptions on it might answer what the correct word is.
    – Rig
    Jan 13, 2014 at 15:51

Sometimes you do need to debug a live system. Yes, you should have a development or staging copy. But there's always going to be differences. This is especially true if the code is running out in the wild on customer hardware. Or potentially, many different customer installations.

I've used the &debugging=1 technique in the past. I suspect most PHP developers have. That flipped a switch in the code that enabled more verbose debugging in the application. That info would usually be dumped to a log file - generally the apache log (using error_log()). But, you can also output it. Our profiler, for example, would gather up information and then output the various benchmarks at the bottom of the page. You can also output the debugging information as an HTML comment, or in a hidden element thats viewable only if you view the page source.

If your site has 'users', you can limit debugging to only a particular user. This way if someone does try to enable debugging, it still wont function unless they are logged in as that user.

Now, you mentioned you were using FileZilla to connect to the server. A developer really should have SSH access to the server. That will make debugging much easier for you. For example, if you were to output your debugging statements to the Apache error log, for example, you could then easily grep that file for your IP address, and see the debugging information generated by your last page request.


Everyone else has already covered the general case:

Formal Validation (/Proven Code): Not feasible for real world programs, though there is a 4000 line OS kernal, that was formally proven, It took many many Russian CS PhD Students many months (please comment if remember the name of this project)

CI Testing << Automated Testing << Testing : make sure you use a coverage tool to check your test cases have 100% coverage.

For your specific case of leaving debug code in production, I have 2 options, both of which require the source code to be recomplied as part of the deployment to a new (Staging/Final Testing) environment.

  1. Remove it at compile time. Wrap debug code in structures like C# and C #ifdef DEBUG to check the build target, (Either Debug or Release) and to automatically remove them at compile time.

  2. Flag it at Deploy time. Put a comment near code that must not be run in the real enviroment. Eg //TODO: Remove This Before Deployment, Then when it is migrated (deployed) to Staging, before compiling the code, run it through a simple script that checks to make sure there are no Flag comments (eg //TODO:) left in the source code.

I suggest the former for if it is longterm and you will want it again (Eg verbose logging mode), and the later if it is a quick hack while you were debugging (Eg various printfs scattered through your code)


As others have said before me, lots of tests (unit and functional), if possible automated and with good code coverage is the key to having a mostly bug-free software.

If I understand the description of your problem well enough, the debugging information is shown on the page served by your server. If that is the case, you should consider putting that information in proper logs on your server. That way it is never shown to the end user, even if you deploy a 'verbose' version to production. This is something that is good to do whatever the project you are working on unless you have very good reasons to do otherwise.


What others say about debugging in production is right. But I have done it sometimes anyway. And I think there are safer ways to do it than yours, though nothing is foolproof.

I don't need such high security myself, I just don't want users to see a bunch of gibberish on their screens.

$debug = true;
$allowed_ips = array('',''); // limit to your own ip's. If your competitor knows them, though, he can spoof it! But if your stuff is not top secret, it's ok. 

if(!in_array($_SERVER['REMOTE_ADDR'],$allowed_ips) ) {
  $debug = false;

if($debug) {
  echo '<pre>' . print_r($some_variable) . '</pre>';

Now users won't see your debugging unless they really want to. If your $800/day depends on keeping this debug information a secret, then don't use the above code. But if the information is not that sensitive, the above will be much better than depending on a GET variable or revealing your secrets to an entire country.

Alternatively, you can create a debug class or function that will check if printing is allowed or not based on your criteria. That's what I do.

Something like this:

class debug {
  public $on = false;
  public static function print($variable) {
    if($on) {
      echo '<pre>' . print_r($some_variable) . '</pre>';

debug::print($some_variable); // class checks if printing is allowed. If not, no printing occurs.

For my program that will make me $800/day (in development), I use apache mod auth to require a password to access any part of the site, as well as restricting debug information to certain IPs. Your question makes me think I should look into better security though.


Actually, I would have two extra validations here. One is the &debug=1 parameter in the request. You can also use some special session variable or cookie setup that only you can activate via a call to a "secret" page (preferably with authentication).

The second validation would be to have in the configuration file, an array with a range of IPs and only calls from an IP in that array can trigger the debugging functionality. something like

In the config:

$debugAllowedIPs = array('', '', /* add more IPs here */);

Have the following method somewhere where it can be accessed globally:

function getClientIp() {
    if (!empty($_SERVER['HTTP_CLIENT_IP']))  return $_SERVER['HTTP_CLIENT_IP'];
    return $_SERVER['REMOTE_ADDR'];

And in your code call something like:

if (in_array(getClientIp(), $debugAllowedIPs)) { /* show debug info here */ }

Please keep in mind that the code in the getClientIp() method is not safe as the value for $_SERVER['HTTP_X_FORWARDED_FOR'] is easily spoofed, however it should serve the purpose of getting the point across for your question.

Not the answer you're looking for? Browse other questions tagged or ask your own question.