I am maintaining a system where a list of string ids are stored against a container (using the term here as a generic entity) object. The database used is MySQL and the table structure looks like this.

|  CONTAINER_NAME     |        ID        |
|          XXYY       |    ffca5e4d7     |
|          XXYY       |    5f0ffca5e     | 
|          AABB       |    0f89d67a6     | 

Each container has around ~200 entries (max <= 300) and there are nearly ~18 million of such containers.

The list of ids for a container is retrieved tens of thousands of times in every few minutes. (The database is shared with different subgroups of the application with extremely high traffic)

Since addition of new containers/ids are far too less compared to the number of reads, I have decided to use a redis instance to lazily cache the container-id lists. The cache key would be the container name and the value is a protobuf serialized java list of strings.

For Java serialization, the proto defintion looks like this:

message Container {
    repeated string id = 1;

However there are containers which do not have any ids associated yet. Instead of looking them up again and again, the status of id association will as well be stored in the proto.

message Container {
    bool hasIds = 1;
    repeated string id = 2;

So the final flow would be like this:

  • When a container is created, put a redis entry with hasIds set to false
  • When a container is searched, query redis
    1. if found
      • if hasIds= false, container does not have any ids yet (db lookup not needed),
      • else read the ids list
    2. if not found, query database
      • if no ids found, put a redis entry with hasIds set to false
      • else cache the list in redis for next lookup
  • When a new id is added (after adding in database)
    1. if found in redis, add to current list

Probable improvement

  • Instead of maintaining the status flag within the proto, a separate container.status key is maintained, so that if container is LRU'ed from Redis, both status and the list is not lost at once.

Please suggest any improvements/modification.

Thank you.

  • Can you actually read a partial value from Redis and would that partial value be a valid protobuf? What is the cost of retrieving a protobuf with an empty list of strings in it compared to keeping a separate boolean to maintain that same information? – Bart van Ingen Schenau Jul 16 '20 at 8:51
  • @BartvanIngenSchenau In this case, if there are ids, the complete collection is always read, there's no partial read required. I do agree with keeping a separate boolean part – DrunkOnBytes Jul 16 '20 at 9:04
  • If there are no ids, do you intend to do only a partial read of hasIds, or would you read the entire Container including the empty list of ids? If the first, I doubt if that is possible. If the second, then hasIds does not add anything that is not already represented by the size of the list of ids. – Bart van Ingen Schenau Jul 16 '20 at 9:25
  • Got your point. Yes, if there are no ids, the full message is still read to get the hasIds – DrunkOnBytes Jul 16 '20 at 9:46
  • Is it not possible for Redis to store an entry with no ids? – user253751 Dec 15 '20 at 14:45

Few things to consider

  1. You said you will load Redis lazily but at some point, Redis might end up having all the keys of MySQL if you don't evict them. You might want to think about the size of Redis to maintain, the eviction strategy in case your Redis shoots over the memory limit, You must set the max memory limit firstly as well or else Redis might crash due to out of memory. Easier still, when you add your key to Redis, use a ttl.
  2. If you use ttl for you key, you might not require hasIds boolean.
  3. How do you plan to load Redis? you say lazily, but you also mention if a new key is added you will save it to Redis. Or are you going to do both? I would prefer a single approach, let the key be brought into Redis only when there is a cache miss and it is being asked for, that way managing your Redis becomes easier, while the keys which are not used would not be there in Redis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.