We follow the cache-first strategy right now.
- Always fetch from the cache first.
- If absent in cache, fetch from DB and update the cache.
- Cache TTL is 30 min. After this, the cache-key expires.
Problem with current setup:
As soon as we upload new data to DB(>10K records), the cached data is stale for us. This stale data is used by Redis users till the keys expire(30 min). Post expiry fresh values will be fetched from DB and set in cache again. We cannot clear these keys since FLUSHALL operation is O(n). Async flushing increases CPU usage and degrades performance. We can also not set this number of keys on upload for obvious performance issues.
Partial Solution 1:
Use logical databases in Redis. Short explanation - One Redis instance has 16 logical databases(
DB15). So, one could segregate their data logically into different DBs. Again, this is just logical. Any blocking operation that is done on any of the DBs blocks operations on other DBs.
How we could use this is by switching b/w 2 logical DBs in Redis. e.g. Project starts with
DB0 as its data source. On upload, we start using
DB1. On the next upload, we come back to
DB0 and so on.
Problem with Partial Solution 1
- If the next upload happens within 30 min, we will switch back to the previous logical DB too quickly. This DB still holds stale keys.
- We have distributed deployment. All the nodes point to the same Redis instance. So, when the upload is handled by let's say Node-1, it switches its logical DB, but other nodes are not aware of this switch and continue to use stale data.
- Using logical DBs is discouraged for various reasons. This thread covers that.
Partial Solution 2:
All nodes use their own dedicated Redis instance and use the logical-DB-switching mechanism.
Problem with Partial Solution 2
- Maintaining multiple instances. Our nodes auto-scale based on the replication factor configured in Kubernetes. How would we create a dedicated Redis instance here?
- Resource consumption.
- If one node processes the latest upload, other nodes(having their own dedicated Redis) will not have this fresh data until their Redis cache keys expire and refresh from DB.
Overall, none of the solutions are fully capable of getting rid of stale data in Cache. Maybe some trade-offs are expected. Maybe some architectural changes are required.
So, I am looking for either the right choices, or best trade-offs, or some other suggestions in the given scenario.