Best strategy to flush Redis cache used in a distributed environment

Question

Current setup:
We follow the cache-first strategy right now.

1. Always fetch from the cache first.
2. If absent in cache, fetch from DB and update the cache.
3. Cache TTL is 30 min. After this, the cache-key expires.

Problem with current setup:
As soon as we upload new data to DB(>10K records), the cached data is stale for us. This stale data is used by Redis users till the keys expire(30 min). Post expiry fresh values will be fetched from DB and set in cache again. We cannot clear these keys since FLUSHALL operation is O(n). Async flushing increases CPU usage and degrades performance. We can also not set this number of keys on upload for obvious performance issues.

---

Partial Solution 1:
Use logical databases in Redis. Short explanation - One Redis instance has 16 logical databases(DB0-DB15). So, one could segregate their data logically into different DBs. Again, this is just logical. Any blocking operation that is done on any of the DBs blocks operations on other DBs.
How we could use this is by switching b/w 2 logical DBs in Redis. e.g. Project starts with DB0 as its data source. On upload, we start using DB1. On the next upload, we come back to DB0 and so on.

Problem with Partial Solution 1

1. If the next upload happens within 30 min, we will switch back to the previous logical DB too quickly. This DB still holds stale keys.
2. We have distributed deployment. All the nodes point to the same Redis instance. So, when the upload is handled by let's say Node-1, it switches its logical DB, but other nodes are not aware of this switch and continue to use stale data.
3. Using logical DBs is discouraged for various reasons. This thread covers that.

---

Partial Solution 2:
All nodes use their own dedicated Redis instance and use the logical-DB-switching mechanism.

Problem with Partial Solution 2

1. Maintaining multiple instances. Our nodes auto-scale based on the replication factor configured in Kubernetes. How would we create a dedicated Redis instance here?
2. Resource consumption.
3. If one node processes the latest upload, other nodes(having their own dedicated Redis) will not have this fresh data until their Redis cache keys expire and refresh from DB.

---

Overall, none of the solutions are fully capable of getting rid of stale data in Cache. Maybe some trade-offs are expected. Maybe some architectural changes are required.
So, I am looking for either the right choices, or best trade-offs, or some other suggestions in the given scenario.

Answer 1

I'm not sure i am understanding your problem correctly. But it sounds like you are simply using redis as an in memory db rather than a cache.

ie you store all the records there all the time. This is simply the wrong way to go about things.

Your client call should only be requesting a small sub set of related records.

It's really hard to talk about this in the abstract, so lets say its Customer data and I'm calling GetCustomerByPostCode I know that for whatever reason, if I get postcode x on one call, i'm likely to get it a few more times within the next few minutes.

Because my database is a bottleneck on this data, I cache the returned customer information for 5 min using the postcode as a key. Any new calls for the same post code in the next 5 min will get the cached response.

Also I have a memory limit on my distributed cache, so if too many different postcodes come in at once, the oldest ones in the cache will be dropped.

Notice that I've chosen a relatively short time to cache the result. This is because I know about the memory limit, the expected repeat call timeframe and the frequency of data updates.

• There's no point in keeping it longer than the expected repeat call, because I am trying to reduce database calls to a level where the db can handle them quickly, not eliminate them completely.

• There's no point in keeping it longer than the expected update frequency, because it will be out of date and need refreshing

• There's no point in keeping it longer than the time it takes for all the caches' memory to be used up, because it will be dropped in any case.

So. every x hours I have some export job which updates all the customers details. When this happens I have out of date cached data for all the postcodes called in the last 5 min.

Now I have a choice. I can accept that, perhaps an average 2.5min of delay for a subset of data is acceptable.

OR I can flush all the keys. Since the number is limited to the postcodes used in the last 5 min there should be far less keys than updated records.

This is how caching should work. If you are putting all the records in the cache, then you might as well not have a cache at all!