How should one debug a PHP web application securely without exposing secrets to competitors?

Question

Recently I made a program. I forget to delete 2 line of codes. That mistake cost me $800 per day every day.

I was programming with PHP. If a visitor uses proxy it redirect somewhere else. Using debugger was impossible because some code contains ioncube. Because the program simply redirect somewhere else no matter what, it's hard to see which part of the code is executed.

So I put a bunch debugging info everywhere. I thought I'll delete them latter anyway.

The most natural way to debug is of course to put debugging info into a file. The problem is I often use proxy. So after I change the program, I often have to download the text file with filezilla. Often the text file does not show what I think it should show. Finally I decided to just display error on the web.

I considered having debugging mode. However, I am afraid I will forget to delete debugging info.

I considered having debugging mode if user do ?debuggingmode=1 for example. However, I was paranoid that somehow my competitor can guess the secret keyword.

I deleted most debugging info. I forget to delete one and that one only show up if users use proxy from the right country. Turns out I do not have proxy from the right country and didn't realize that. After the program works for 24 hours, I uploaded that to my main domain.

My competitor, using proxy, see the debugging code. He copy the idea and that's how I lost $800 per day.

In retrospect, I really have a hard time seeing where I went wrong. I have been super careful. Yet it happened.

How should one debug a PHP web application securely without exposing secrets to competitors?

Answer 1

I really have a hard time seeing where I went wrong

The major mistake was that you reinvented the wheel. Instead of using default mecanisms for logging, you invented your own, which displayed the information within the page. A logging framework would rather store logs in log files, letting you to consult those logs later by SSHing to the server.

As for the bugs, producing bug-free code implies using specific techniques such as formal proof. Given their expensiveness, those techniques are appropriate for life-critical applications such as applications which control aircraft traffic or space shuttles, but are an overkill for nearly every business application.

■ See They write the right stuff in Fast Company magazine.
_{The article describes the methodology used at NASA, as well as the cost of producing software this way.}

■ See Mechanizing Proof (Mackenzie 2004).
_{The book summarizes the history of automated proof of software, explaining the pros and cons of such proof, as well as the reasons it's not commonly used by businesses to write reliable software.}

This being said, there are a bunch of techniques used for business applications to ensure software quality. Those includes but are not limited to:

• Informal code reviews,
• Formal code inspections,
• Testing,
• Personal desk-checking of code,
• etc.

■ See Code complete (McConnell 2004), Programming Productivity (Jones 1986a), Software Defect-Removal Efficiency (Jones 1996), and What We Have Learned About Fighting Defects (Shull et al. 2002).

Also, don't forget continuous integration and continuous delivery. It helps in automatically rolling back the app in production to a working version when a revised one appears to have an issue which was missed during code reviews and unit testing, but caught once the app is deployed.

■ See The Secret to Safe Continuous Deployment (video)
_{It explains what techniques were set up at Google to prevent bugs which couldn't be found before deployment from staying for too long in production. It also describes pdiff and how was it used to catch bugs, including ones which were unrelated to presentation layer.}

Answer 2

You should never debug in production.

You should always have a test environment which is identical to the production environment and debug there.

When you need to change code in the test environment (like for adding debugging statements), you should make sure that they don't go into production.

A professional setup usually looks like this:

Production
   ^
Staging
   ^
Development

The "Production", "Staging" and "Development" instances of your application should be as identical as possible so that a bug which occurs in "Production" can be reproduced in "Staging" and "Development", but still be completely separated from each other so that whatever happens in one of the instances doesn't affect the others.

When you need to analyze a problem, you do so in "Development". Mess around with debug statements and experiment all you want. When you found a solution, you apply that fix to the unchanged codebase in "Staging" and verify that the fix works. Then you promote the fix to "Production".

A proper version control and build management system can help you with that.

Answer 3

This is rarely done because the effort is not worthwhile. Even if you lose $800 a day, the effort of proving a program correct quickly becomes larger than that, which implies that there is no business case for doing it.

If being certain is worth that much (e.g. for Space Shuttle software or missile control), then you perform formal verification, exhaustive testing of all possible inputs, etc. Granted, it's also extremely difficult as well as slow and expensive. But projects with billion-dollar budgets also tend to have extremely bright people on them. (Or maybe they just used to - present-day headlines seem to contradict that rule.)

Answer 4

Sometimes you do need to debug a live system. Yes, you should have a development or staging copy. But there's always going to be differences. This is especially true if the code is running out in the wild on customer hardware. Or potentially, many different customer installations.

I've used the &debugging=1 technique in the past. I suspect most PHP developers have. That flipped a switch in the code that enabled more verbose debugging in the application. That info would usually be dumped to a log file - generally the apache log (using error_log()). But, you can also output it. Our profiler, for example, would gather up information and then output the various benchmarks at the bottom of the page. You can also output the debugging information as an HTML comment, or in a hidden element thats viewable only if you view the page source.

If your site has 'users', you can limit debugging to only a particular user. This way if someone does try to enable debugging, it still wont function unless they are logged in as that user.

Now, you mentioned you were using FileZilla to connect to the server. A developer really should have SSH access to the server. That will make debugging much easier for you. For example, if you were to output your debugging statements to the Apache error log, for example, you could then easily grep that file for your IP address, and see the debugging information generated by your last page request.

Answer 5

Everyone else has already covered the general case:

Formal Validation (/Proven Code): Not feasible for real world programs, though there is a 4000 line OS kernal, that was formally proven, It took many many Russian CS PhD Students many months (please comment if remember the name of this project)

CI Testing << Automated Testing << Testing : make sure you use a coverage tool to check your test cases have 100% coverage.

For your specific case of leaving debug code in production, I have 2 options, both of which require the source code to be recomplied as part of the deployment to a new (Staging/Final Testing) environment.

1. Remove it at compile time. Wrap debug code in structures like C# and C #ifdef DEBUG to check the build target, (Either Debug or Release) and to automatically remove them at compile time.

2. Flag it at Deploy time. Put a comment near code that must not be run in the real enviroment. Eg //TODO: Remove This Before Deployment, Then when it is migrated (deployed) to Staging, before compiling the code, run it through a simple script that checks to make sure there are no Flag comments (eg //TODO:) left in the source code.

I suggest the former for if it is longterm and you will want it again (Eg verbose logging mode), and the later if it is a quick hack while you were debugging (Eg various printfs scattered through your code)

Answer 6

As others have said before me, lots of tests (unit and functional), if possible automated and with good code coverage is the key to having a mostly bug-free software.

If I understand the description of your problem well enough, the debugging information is shown on the page served by your server. If that is the case, you should consider putting that information in proper logs on your server. That way it is never shown to the end user, even if you deploy a 'verbose' version to production. This is something that is good to do whatever the project you are working on unless you have very good reasons to do otherwise.

Answer 7

What others say about debugging in production is right. But I have done it sometimes anyway. And I think there are safer ways to do it than yours, though nothing is foolproof.

I don't need such high security myself, I just don't want users to see a bunch of gibberish on their screens.

$debug = true;
$allowed_ips = array('192.168.0.220','173.46.89.255'); // limit to your own ip's. If your competitor knows them, though, he can spoof it! But if your stuff is not top secret, it's ok. 

if(!in_array($_SERVER['REMOTE_ADDR'],$allowed_ips) ) {
  $debug = false;
}

if($debug) {
  echo '<pre>' . print_r($some_variable) . '</pre>';
}

Now users won't see your debugging unless they really want to. If your $800/day depends on keeping this debug information a secret, then don't use the above code. But if the information is not that sensitive, the above will be much better than depending on a GET variable or revealing your secrets to an entire country.

Alternatively, you can create a debug class or function that will check if printing is allowed or not based on your criteria. That's what I do.

Something like this:

class debug {
  public $on = false;
  public static function print($variable) {
    if($on) {
      echo '<pre>' . print_r($some_variable) . '</pre>';
    }
  }
}

debug::print($some_variable); // class checks if printing is allowed. If not, no printing occurs.

For my program that will make me $800/day (in development), I use apache mod auth to require a password to access any part of the site, as well as restricting debug information to certain IPs. Your question makes me think I should look into better security though.

Answer 8

Actually, I would have two extra validations here. One is the &debug=1 parameter in the request. You can also use some special session variable or cookie setup that only you can activate via a call to a "secret" page (preferably with authentication).

The second validation would be to have in the configuration file, an array with a range of IPs and only calls from an IP in that array can trigger the debugging functionality. something like

In the config:

$debugAllowedIPs = array('127.0.0.1', '192.168.0.1', /* add more IPs here */);

Have the following method somewhere where it can be accessed globally:

function getClientIp() {
    if (!empty($_SERVER['HTTP_CLIENT_IP']))  return $_SERVER['HTTP_CLIENT_IP'];
    if (!empty($_SERVER['HTTP_X_FORWARDED_FOR'])) return $_SERVER['HTTP_X_FORWARDED_FOR'];
    return $_SERVER['REMOTE_ADDR'];
}

And in your code call something like:

if (in_array(getClientIp(), $debugAllowedIPs)) { /* show debug info here */ }

Please keep in mind that the code in the getClientIp() method is not safe as the value for $_SERVER['HTTP_X_FORWARDED_FOR'] is easily spoofed, however it should serve the purpose of getting the point across for your question.