Pretend for a moment that something catastrophic happened; you're hacked, and your production database becomes a mess.
How far (time) should you be from your rolling out your latest backup and resuming operations as normal?
Of course, "immediately" is "ideal", but I'd like to see some real answers. Perhaps some existing, real-life situations that you're currently in with regards to backups.
For databases, it should be trivial to roll out your latest backup. It's purely a question of how large your database is, as to how long it takes to restore. If it's a huge database, it may take hours, but that's life. If the bosses don't like it, they can spring for a second server.
If you've had a real catastrophe, you should be able to get to your offsite backup in no more than an hour. If it takes longer than that, you might want a new offsite strategy.
If it's a hack, rather than a hardware problem, it's more important (and time consuming) to make sure you don't get hacked again. That's the time sink. Identify how they got you, and close the hole. If you don't do that, then you're just wasting your time.
For code bases, you should be able to deploy that extremely quickly. It should be just a matter of re-deploying the production code, which, hopefully, is automatic or close to it. If the deployment is largely manual, do something about that. I once dealt with this awful .Net package that was all dlls that had to be manually registered, and they weren't all backed up in the same place. Nightmare.
The most important thing to do is to practice. Bring your people in on the weekend and do a bare metal rebuild. Then take 'em to a bar for a critique. Everyone should know where everything is, what the plan is, and what their part in the plan is. If you have to make a decision during a crisis, you've failed. It should be automatic.
I'm asking how long it should take to restore to a backup.
This is actually a political question, because the shorter the time the more money it will cost.
If your business cannot afford downtime, you must plan accordingly and have hot spares (computers started, ready to take over). This can either be physical machines or vmware instances "elsewhere".
If you can afford 1 hour downtime, you can run inside vmware with your servers, and have regular snapshots which you can then restore in another vmware host and continue.
If you can afford 8 hour downtime, you can restore from tape to a cold spare.
If you can afford 24 hour downtime, you can get the tape from another physical location and buy a new computer to reinstall to.
Also note that the backups must be recent for this to work. This again requires good planning to ensure that they are always good enough.
And, do fire drills! If it is important to your business, you should practice to ensure that you can do this correctly when disaster strikes.
It depends on how much money the company will be out from the system being unavailable. In some large, or mission critical, companies... you just can't go down. There must be a failover plan, or multiple ones. I can't really speak to that, but I'm sure some people over at serverfault can (this question would probably do much better over there actually).
I can speak to a smaller, more budget restrained company though. I worked for (small, 10 people) company a few years back and for us it was 2 hours until normal working order was considered by management to be reasonable, and the backup was generally never more than 4 hours old. Any longer than that, or further back for the backup, then the company started losing money. Maintaining backups that occurred more often than that would cost more than the loss.
In this case, it also depended on which backup: on-site or off.
Off site backups took longer, and were backed up less frequently (once a day, nightly), but they were only in place in case the building caught fire or something. We never had to use those, thankfully, and I forget what the time limit on being back up and running was - but it was certainly on the order of days if I recall.
On site backups were just dumps of the datastore(s) and the shared folders on the server. Coming back online meant restoring the data to a different machine in the office (my dev machine actually... we were on a budget) until the server was fixed - this usually took 15 minutes tops. People could keep working. After the server was fixed, the 'system' was taken offline while the data was dumped from my machine, backed up and restored back to the server. The server was then brought back online, and we were ready to go. That usually took 30-40 mins (after the server was fixed). This happened twice in the 2 or so years I was there. Once was a heating issue in the server closet, and we waited for the A/C to get fixed.
A standby server is always the best choice - and keep switching between the master and standby as a weekly/monthly operation to ensure there are no snags (Like the standby can't take the peak load of the site for example.).
If you are going to tape, it is very imperative you test often because schemas change / developers forget stuff and lo-behold, you have to write code that will parse and migrate from old data instead of a direct restore. That is the worst case situation.
