1

As a System Architect I need to deploy a resillient database server in an isolated private cloud. The cloud is OpenStack with Ceph backend and the Database is SQL.

Is a single database server VM with a local backup sufficient or do I need a pair of database server VMs in active / standby hosted apart so that they don't share the hardware fault? The answer must consider the likelihood that a single hardware fault (eg CPU) destroys a running SQL server VM and corrupts both the primary and backup databases preventing automated recovery.

Assumptions:

  1. This is NOT about load balancing nor hardening - the load is light and the network is isolated. It IS about automatic recovery from hardware faults.
  2. 10 minute outage is OK, 1 hour is not.
  3. OpenStack Instance HA will automatically create a replacement SQL Server VM away from the hardware fault and mount the database and continue providing service unless both primary and backup databases are corrupt.
  4. The primary and backup databases are each on dedicated virtual volumes, with resillient Ceph backend, so if one filesystem is corrupted the other may not be.

Why do I care?
a) The active / standby is additional complexity b) Having an extra VM consumes some of the finite CPU and RAM from the private cloud

Thanks to all who participate for your time and energy.

  • 1
    You don't need high availability because you can tolerate short outages. So keeping a standby DB running is a waste of resources. Still, you will want to ensure that there is reserved capacity, that you can easily and quickly restore DB state (test this at least every day), and test automatic failover during planned outages. – amon Jul 3 at 6:39
0

As short outages (long enough to restart a VM) are acceptable, you don't need the immediate failover capability of an active/standby configuration.

That makes the main concern that you don't get a corruption of the database storage. You have partially tacked that by having redundant storage with a main database and a backup database.

What still can go wrong is that your single database server is performing a write action to both the main database and the backup database at the same time when a CPU fault occurs.

It might be that the database server is designed to be crash-resistant in the sense that a crash of the server will never leave the storage in an unrecoverable state. If the software is designed for that, then you are good to go.

If the software can leave the storage in an unrecoverable state, then you should ensure that there are never concurrent write actions to both the main and the backup database. Maybe this can be configured in the database software itself, but otherwise you would have to do so in your own application by using separate, sequentially executed, write actions to each database.

| improve this answer | |
  • The likelihood of corruption of both backup and primary databases seems incredibly unlikely. Also the backup database can't be automatically mounted allowing service to continue. – Adam Ratcliff Jul 11 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.