How do companies keep websites hidden when in development?

Question

I am new to this and also a new 19 year old PHP developer that just got hired so I am unsure as to how this works. Many companies when developing their websites keep their sites hidden from being indexed. What are some ways html5/css and then php/mysql work is hidden from search engines? If I am not wrong, these are the techniques:

1. Offline development: using local storage to render html/css; not sure how PHP/mysql would work with that.

2. Using .htaccess to prevent access

3. Using VPN to prevent access.

Answer 1

The best practice is to have an approximation of the production stack on your local developer machine. This typically includes the database, web server and your customized code. Do all of your development there. Never edit code on a production server.

When your business stakeholders are ready for the functionality to go live, push your code from your local machine to the production server. You might use a test server so internal stakeholders can preview your changes before you go live.

You should be sure to use a version control system to manage your code. Git and Mercurial are solid options.

Answer 2

Merely asking search spiders to please not index your site is a very weak defense, if any.

The usual way to go about this is as follows:

• While in development, you run the code on a development machine that isn't accessible from outside your LAN, or possibly not even from anywhere except the same machine (localhost). This is usually done with a firewall, either at the network level (an actual LAN firewall; if your company is worth anything at all, you have one) or locally (a 'personal' firewall). In addition, you can configure your local apache server to only accept connections from localhost (127.0.0.1).
• The acceptance version (i.e., a sandboxed version for the customer to try out and test) runs on a machine that is identical to the production environment (as much as possible - some configuration details will have to be different); you limit access similar to how you did with the development environment, except you also grant access to the customer's network. If this isn't possible (e.g., because the customer doesn't have their own dedicated IP, or because they can't be bothered figuring out what it is), you may consider adding HTTP authentication and giving the customer a password; this can be done at the web server level (i.e., Apache), with one caveats: HTTP authentication goes over the wire unencrypted, so if the acceptance version contains anything remotely confidential, serving the site over HTTPS exclusively is an absolute must.

As to how you'd set these things up: If you run a decent *nix distribution, it comes with a package manager (apt, port-install, rpm, etc.); use this to install the components you need, and read the documentation. Ask a sysadmin for help if you can't figure it out yourself.

Answer 3

For polite crawlers (ie. any major search engine), you can offer a /robots.txt file with:

User-agent: *
Disallow: /

Of course that is only advisory, and won't disallow people from actually visiting the site if they know the URL. A better approach is to put the site behind a login of some kind if you need access from the internet at large, or simply host the server on an internal web server.

Answer 4

In the spirit of "going in with more knowledge":

Html/Css can be hidden with robots.txt, but be sure to change those files when you push the site to production. robots.txt does not prevent some random person guessing the development website (if it is visible to the whole world).

.htaccess could be used to add a username/password combo to any site being developed.

Here is an example of how you could use a VPN:

All requests on the VPN go through a firewall. That firewall (running some software like pfSense) examines website requests and does some redirection based on url. For example, the url http://example.com.staging will direct to the version of example.com on your staging server, and the url of http://example.com.development will direction to the version of example.com on your development server.

Since all of this is on a private VPN, none of these websites are accessible from the outside world.

Also, to clarify some things with your original question. The php/mysql work is being done server side, so web crawlers, browsers, etc. never see the code--they only see the output of the code (that is, if you've configured your server correctly and aren't rendering .php files as text).

I'm sure I've misread your question, but I found your use of the phrase "local storage" ambiguous. LocalStorage can also refer to an HTML5 technology for storing assets locally on a client's machine. It is not used for rendering html/css, it is only a datastore (the browser renders the html/css). Local storage typically has nothing to do with how companies keep websites hidden in development.

PHP files can be viewed locally if you are running a web server on your machine and the files are in the appropriate directories (usually the htdocs directory if running Apache).

Answer 5

Search engines work using spiders: scripts that crawl the web from one link to another. If your site is new and no links to it exist from elsewhere, nothing's going to find you. That, plus a robots.txt file configured to keep any spider that does somehow wander across your site from perusing it, should be enough.