Removing Redis Entries from Multiple Redis Instances

Question

I have an interesting problem that I'd like to solve. Let's say I have 2 deployments, US Data Center and Europe Data Center each containing Web Servers and a Redis Instance (replicated, but could eventually grow into its own cluster using TWEMPROXY).

We have a Load balancer that identifies the closest available Data Center and routes the requests to it. Our application logic can identify a Tenant and route the request to the appropriate data center irrespective of where it is called from.

For e.g. if I am accessing US Tenants from Europe, the Load Balancer will send the Requests to the Europe Data Center, but the application will identify the tenant and access the data from the Tenant's database (which in this case resides in the US).

While serving requests, the data is also cached in paired Redis Server (which resides in Europe). And that's when problems start.

When the data is invalidated by someone in US, the data is cleared off from the paired US redis servers, but not from Europe. So if someone accessed the US data from European servers, they will continue getting stale data unless it is cleared off.

Is there any way to invalidate an entry in the US Data Center and then also invalidate it from the Europe Data Center? Has anyone implemented a strategy whereby you can easily identify the Redis Servers where the data being invalidated resides so you can selectively purge it from servers where it resides? I'm wondering if my only option is to send a message to every Redis Server to clear the key off if it exists. This is highly impractical and will not scale.

The other alternative is to always access and cache Europe data from European Redis Caches irrespective of where the request comes from. This might increase the Redis Latency for someone who is from US and is accessing the Euro Data Centers, but the chances of this happening are pretty low.

Would love to hear what others think about this.

Anup

Answer 1

Two things in CS are hard: naming, cache invalidate, and off-by-one.

For N=2 deployments, your naive approach sounds suitable:

send a message to every Redis Server to clear the key off if it exists. This is highly impractical and will not scale.

I don't agree that it is so impractical. Use small identifiers (not GUIDs), batch them up, and every K milliseconds send the batch of invalid IDs to the other datacenter. Flooding invalidation messages gets ugly as N grows large, but 2 is not large.

Here is a more sophisticated approach:

Rather than letting clients perform Read operations as they do now, make them perform a (higher overhead) Lease operation against your data source. Client submits an expiry time, a few seconds or hours into the future. Source appends client name and expiry time to metadata for the retrieved value. Client may freely enjoy local cache hits of the cached data up to the expiry time, after which they will suffer cache misses. Source will send cache invalidate messages to all lease holders, and clear that list, upon receiving an update.

This makes each access to the data source a bit more expensive as we maintain metadata, but it eliminates the wasted bandwidth of flooding cache invalidate messages to clients who (A) are potentially caching an item but (B) in reality the probability it is cached is extremely low. Some items will be hot, others cold. To retain a hot item indefinitely, a client can notice that a cache hit happened after half the lease interval elapsed, satisfy the hit at once, and queue up another query (a lease renewal) against the original data source, to extend the expiry timestamp. During scheduled server maintenance or reboot, the server can decline read requests until all clients expire, or better it can clear all leases by sending cache invalidates to all clients appearing on all metadata lists.