I am new to this and also a new 19 year old PHP developer that just got hired so I am unsure as to how this works. Many companies when developing their websites keep their sites hidden from being indexed. What are some ways html5/css and then php/mysql work is hidden from search engines? If I am not wrong, these are the techniques:

  1. Offline development: using local storage to render html/css; not sure how PHP/mysql would work with that.

  2. Using .htaccess to prevent access

  3. Using VPN to prevent access.

  • 1
    The best solution, when possible, is always to have a test server completely identical to the production server. And even in this case there can be unexpected surprises...
    – Avio
    May 21, 2012 at 10:08
  • If you want to do full local php development take a look at easyphp.org it's simple installer that will get Apache Web Server, MySQL db and PHP installed and configed on your windows desktop. May 21, 2012 at 12:54

5 Answers 5


The best practice is to have an approximation of the production stack on your local developer machine. This typically includes the database, web server and your customized code. Do all of your development there. Never edit code on a production server.

When your business stakeholders are ready for the functionality to go live, push your code from your local machine to the production server. You might use a test server so internal stakeholders can preview your changes before you go live.

You should be sure to use a version control system to manage your code. Git and Mercurial are solid options.

  • Hey greg, is there a link you can point me to that shows me how to do these things? all coding i know is from learning it myself and so first time i am working at a company though they said it will be a learning environment for me and they are ok with that but i want to go in with as much knowledge as possible. Would really appreciate it.
    – ariel
    May 21, 2012 at 3:40
  • This pattern is called a "private workspace". The site informit.com/articles/article.aspx?p=30350 seems to have a reasonable overview.
    – greg
    May 21, 2012 at 11:56
  • Also you will probably never find an exact recipe for your situation. You should be able to identify each of the subsystems in your production environment and find installation help for getting them on your local workstation.
    – greg
    May 21, 2012 at 11:58
  • And while you are at it, source control your datbase as well!
    – HLGEM
    May 21, 2012 at 17:49
  • In one of my previous jobs our automated build process applied the entire history of data migrations to a gold data set (a subset of production data) by applying a sequence of SQL files in succession.
    – greg
    May 22, 2012 at 2:13

Merely asking search spiders to please not index your site is a very weak defense, if any.

The usual way to go about this is as follows:

  • While in development, you run the code on a development machine that isn't accessible from outside your LAN, or possibly not even from anywhere except the same machine (localhost). This is usually done with a firewall, either at the network level (an actual LAN firewall; if your company is worth anything at all, you have one) or locally (a 'personal' firewall). In addition, you can configure your local apache server to only accept connections from localhost (
  • The acceptance version (i.e., a sandboxed version for the customer to try out and test) runs on a machine that is identical to the production environment (as much as possible - some configuration details will have to be different); you limit access similar to how you did with the development environment, except you also grant access to the customer's network. If this isn't possible (e.g., because the customer doesn't have their own dedicated IP, or because they can't be bothered figuring out what it is), you may consider adding HTTP authentication and giving the customer a password; this can be done at the web server level (i.e., Apache), with one caveats: HTTP authentication goes over the wire unencrypted, so if the acceptance version contains anything remotely confidential, serving the site over HTTPS exclusively is an absolute must.

As to how you'd set these things up: If you run a decent *nix distribution, it comes with a package manager (apt, port-install, rpm, etc.); use this to install the components you need, and read the documentation. Ask a sysadmin for help if you can't figure it out yourself.


For polite crawlers (ie. any major search engine), you can offer a /robots.txt file with:

User-agent: *
Disallow: /

Of course that is only advisory, and won't disallow people from actually visiting the site if they know the URL. A better approach is to put the site behind a login of some kind if you need access from the internet at large, or simply host the server on an internal web server.

  • 1
    This and Greg (...the other Greg)'s answer sum everything up pretty much perfectly.
    – Anonymous
    May 21, 2012 at 6:44

In the spirit of "going in with more knowledge":

Html/Css can be hidden with robots.txt, but be sure to change those files when you push the site to production. robots.txt does not prevent some random person guessing the development website (if it is visible to the whole world).

.htaccess could be used to add a username/password combo to any site being developed.

Here is an example of how you could use a VPN:

All requests on the VPN go through a firewall. That firewall (running some software like pfSense) examines website requests and does some redirection based on url. For example, the url http://example.com.staging will direct to the version of example.com on your staging server, and the url of http://example.com.development will direction to the version of example.com on your development server.

Since all of this is on a private VPN, none of these websites are accessible from the outside world.

Also, to clarify some things with your original question. The php/mysql work is being done server side, so web crawlers, browsers, etc. never see the code--they only see the output of the code (that is, if you've configured your server correctly and aren't rendering .php files as text).

I'm sure I've misread your question, but I found your use of the phrase "local storage" ambiguous. LocalStorage can also refer to an HTML5 technology for storing assets locally on a client's machine. It is not used for rendering html/css, it is only a datastore (the browser renders the html/css). Local storage typically has nothing to do with how companies keep websites hidden in development.

PHP files can be viewed locally if you are running a web server on your machine and the files are in the appropriate directories (usually the htdocs directory if running Apache).


Search engines work using spiders: scripts that crawl the web from one link to another. If your site is new and no links to it exist from elsewhere, nothing's going to find you. That, plus a robots.txt file configured to keep any spider that does somehow wander across your site from perusing it, should be enough.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.