How can adding redunancy adversely affect performance

Question

I have been reading about system design and was going through this popular article - A Word on Scalability

The author here says:

An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance.

and then again:

It requires applications and platforms to be designed with scaling in mind, such that adding resources actually results in improving the performance or that if redundancy is introduced the system performance is not adversely affected

I can't quite understand how, or in what case could adding redundancy affect performance adversely.

Answer 1

Scenario A: You have a system that solves problem X. Scenario B: You have a system that solves problem X and has to ensure that it is always synchronized with the redundant backup.

It is pretty clear that the system in Scenario B has to perform strictly more work than the system in Scenario A, therefore the redundancy has a performance impact.

However, note that this is not necessarily always the case. It depends on how the redundancy is designed, what it is designed to protect from, and so on.

An example that is probably familiar to many of us: RAID 1 vs. RAID 5. RAID 1 achieves redundancy by mirroring. This means everything you write to disk is also written to all other disks as well. So, the write performance is the minimum of the individual write performances of all drives. But reading can be done in parallel (assuming that you are not actually in the disaster case at present), so the read performance is actually the sum of the individual read performances of all drives.

Oh, but wait: it actually isn't. Not quite, at least: because if you rely on that fact for your system to work, then it is no longer redundant! As soon as you lose one drive, your read performance will go down. So, your system has to be designed to be able to work with that performance. If it is designed to work with being able to spread the reads across all drives, then it is no longer redundant. And if you have 7 mirrored drives, but your system requires the read performance of 6 drives, then your RAID is actually only 1-redundant, not 6-redundant as you would a 7-drive RAID 1 normally expect to be.

So, while it turns out that RAID 1 does not necessarily have a negative performance impact, and in fact, can even boost performance, actually relying on this performance boost will negate the redundancy.

RAID 5, OTOH, spreads out both writes and reads across all drives and uses an EDAC code (error detection and correction) to be able to reconstruct data. This means you have the slight performance overhead of computing (or reconstructing) this EDAC code, but OTOH, you get a performance boost also for writes. But again, as soon as you are in the failure case, your performance drops, and if you rely on the performance, then you don't have redundancy.

So, RAID 1 is an example of a redundancy scheme that has practically no negative overhead in the worst case, and can speed up one of the two operations (read but not write) in the best case. RAID 5 is an example of a redundancy scheme that has a slight overhead in the worst case (reconstructing the lost date from the EDAC code), but speeds up both read and write in the best case.

Answer 2

Two simple examples:

1. Adding an index to a database table is a common way to introduce redundancy, with the intention of speeding up read operations on that table. However, if a table is more frequently used for write operations than for read operations, adding an index might slow operations down - each write operation now has to update the index.

2. A second example for redundancy are backups. Backups usually don't speed up anything, they always need additional CPU time.

Answer 3

First, as others have pointed out, in general doing more, costs more, so in most cases adding more work results in an increased cost (aka performance reduction).

Secondly, you are missing the most important part of that sentence.

An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance

Emphasis mine.

It’s not that redundancy results in a loss of performance that is important, since occasionally you may be able to get the redundancy for free or even at a reduced cost (aka increase in performance). The important part is that adding resources must counteract any additional cost. IOW, you can must be able to get more done by adding additional resources.

Here’s a (non-realistic, example), a web farm connected up to a single SQL Server database. You want to provide redundant access (more servers in case one of them fails). You have 32,765 servers connected to this database. Each web server has a long running connection to the database and opens up a new temporary connection upon each request. Say this is working just fine, request come, get routed to a web server, which opens up a connection, gets an answer and then closes the connection. Now, you decide to add 1 more web servers and suddenly it’s either failing entirely or taking significantly longer, what happened? SQL Server only allows 32,767 connections, when you add the additional web server for additional redundancy, because of the permanent connection, you only have one free connection at any one time available for use. Increasing resources available drops your performance.

The reason why the example is rubbish is because long before you run out of connections because of too many computers, you’ll run out of connections because of too many users. You aren’t going to have one server per user, you’re going to have thousands of users and connections per server, and so your number of simultaneous users will be your limiting factor. But the underlying problem is the same, and you can’t solve your problem of having too many too many users using too many connections by adding more web servers.

Scalability is about identifying what resources to add, and how to do so without that slowing things down.