I recently started working on a Spring web application, in which a cache for some heavily used data is stored in the JVM of each node in the AWS cluster. Any change in any of the node values results in a REST call to all the nodes to refresh the cache in the JVM. There are many issues in this design, mainly:
The Collection used for cache is HashMap, which is not thread safe
The info of the nodes is stored in the DB and this is also stored in this cache. Any cache miss will result in skipping the node altogether resulting in stale values.
Insecure REST calls among nodes to refresh the cache.
The source of truth is the cache, not the DB. In case of any cache misses the DB is not referred.
IMO, the ideal approach is to used a centralised cache like EHCache or Redis or use something like Hazelcast. What are the other ways in which such a problem is solved?