I have an interesting problem that I'd like to solve. Let's say I have 2 deployments, US Data Center and Europe Data Center each containing Web Servers and a Redis Instance (replicated, but could eventually grow into its own cluster using TWEMPROXY).
We have a Load balancer that identifies the closest available Data Center and routes the requests to it. Our application logic can identify a Tenant and route the request to the appropriate data center irrespective of where it is called from.
For e.g. if I am accessing US Tenants from Europe, the Load Balancer will send the Requests to the Europe Data Center, but the application will identify the tenant and access the data from the Tenant's database (which in this case resides in the US).
While serving requests, the data is also cached in paired Redis Server (which resides in Europe). And that's when problems start.
When the data is invalidated by someone in US, the data is cleared off from the paired US redis servers, but not from Europe. So if someone accessed the US data from European servers, they will continue getting stale data unless it is cleared off.
Is there any way to invalidate an entry in the US Data Center and then also invalidate it from the Europe Data Center? Has anyone implemented a strategy whereby you can easily identify the Redis Servers where the data being invalidated resides so you can selectively purge it from servers where it resides? I'm wondering if my only option is to send a message to every Redis Server to clear the key off if it exists. This is highly impractical and will not scale.
The other alternative is to always access and cache Europe data from European Redis Caches irrespective of where the request comes from. This might increase the Redis Latency for someone who is from US and is accessing the Euro Data Centers, but the chances of this happening are pretty low.
Would love to hear what others think about this.