Do application-level caches belong inside web servers?

Question

A coworker and I have a disagreement about where the cache for a webserver should go. We currently have it implemented as a global (static) in-process cache, which I don't agree with. I think the cache should be offloaded into another server using something like redis. I've always been under the impression that global state is bad, and particularly bad in web servers. State belongs in databases. However, my coworker brings up a fair point which is that offloading it introduces the overhead of network transfer for every cache hit. I can't argue with that, but in-process cache just seems bad. I know we'll eventually run into concurrency issues with the global cache.

Do application-level caches belong inside of web servers, and under what circumstances is this a good choice?

Edit: Some information about what is being cached: In this case we have a microservice that transforms some data from another service before it's served to clients. The reason for caching is two-fold: 1) to reduce the load on the source service, and 2) to cache the results of the data transformation.

Answer 1

As always, one must differentiate: what information is to be stored in the cache?

I always go with these simple rules:

• Information every webserver instance can calculate for itself should go into an in-RAM cache as these need to be accessible ASAP on request but have no need to be shared (never change or do not contain information relevant to other servers).
  • results of complex calculations that every server can perform independently
  • immutable, shared state (e.g. DB results that do not change: top-10 users of the previous week, ...)
  • Code-Metadata caches (e.g. Annotations or calculations resulting thereof)
• Shared state that must be accessible faster than your DB can do should go into a separate cache server [cluster] (e.g. Redis, Memcached, ...). Note that many of these things can be tackled by an intelligent load balancer.
  • IP and User ban information
  • Session State (load balancers can handle this using affinity cookies if the application architecture allows it)
  • queues, domain events
• Shared state that needn't be accessible faster than your DB can do must stay in the DB. Using caches for things that needn't necessarily be cached is a useless source of errors and failures.

As to the question edit: IMHO this data fall in the first category: Every server can calculate the transformation for itself. However, if we are talking about gigabytes of data here, i'd suggest that you do the transformation once, store the result in your DB and only cache the parts requested most frequently. You could also put another 16GB of RAM into your servers... Probably cheaper than implementing several cache layers.

Answer 2

If what you are after is the elimination of global state in your application then the static keyword definitely sucks to have in your code (you say you don't like it, so I'm going to assume you know the disadvantages the static keyword brings to application development).

Considering the cache issue, whether it is in-process or out-processed to a system such as Redis, it would not really matter what exactly you were using, be it memory, file or Redis, as long as the cache layer was static, everything would really stay the same (it would be global).

Needing a good caching layer is definitely not a trivial problem. During my becoming of hopefully a better programmer I have gone through about 5 versions of cache layer interfaces (which either I or a team I was on coded), before finally arriving to a solution which I liked.

But even then, having well documented caching layer (a static one aside, because that's a thing I try to avoid in my applications) is not a solution to a problem but merely a mean how to achieve it.

The problem is something entirely different, that is my application is slow, I want to cache certain data and I want to do it efficiently based on the design of my application.

The two problems which occur most often when you are considering creating a caching layer are the following:

• Do I want to share cache among processes?
• Do I worry about having thousands of processes, each having its own cache poluting the memory pool?

There are many different types of applications: standalone apps the client installs, web applications, web services (REST, SOAP), web presentations,...

For all these a caching layer may be considered as a possibility to make the application faster, but there is a problem, the applications are completely different and for such different approaches must be taken when trying to design cache.

---

Cache for web services

Web services, REST, SOAP, usually have one task. Abstract direct connection to a database so clients can consume the database data without you being forced to expose your data source. On top of that middle layers may be added (such as balancing or cache in question) to improve the performance (usually calculated from response time).

Whether you want an in-process cache or an outsourced one (file/Redis, does not matter), depends on the design of your application.

Consider a PHP web service. PHP works in a way that for each request a new instance of an application is made. This means, if you cached data in-process, the cached data would be only available inside the single request, which could speed up your application a little, but if another endpoint consumed the same data as the endpoint which was first hit, the second endpoint would need to load the data all over again, because it couldn't access the cached data from endpoint one process.

Naturally, you'd want to share the cached state, therefore you'd have only one option. Either make your own caching system, most likely using files, or use a system like Redis, so different processes can access the same data and don't need to hit the database.

On the other hand, if you had a web service, which would have only one instance and memory was shared between endpoints, perhaps in-process cache would actually be a better alternative, considering multiple endpoints would still be able to access it.

---

Your colleague is afraid of performance hit because you need to go to the Redis server rather than directly accessing memory. Believe me, that is really the slightest of problems. The performance drop is so little that it is still better sharing state of database using this way than having to query the database directly.

You can define how you want to store the data in Redis, choose the best data type for your problem. It's quite obvious, traversing a binary search tree for a value combination based on some hash will be faster than having to query database and compose the data inside.

Yes, databases have caches themselves, but even then it's either:

1. query db -> db calls its cache -> db gets data from cache -> db returns data
2. query cache -> get data

Fewer steps usually means the operation is going to be faster and in this case it is.

---

You should cache into outsourced cache things which you know will be shared among requests (data from database is usually shared). For how long and how much data you want to actually cache is up to you and your business rules.

You should cache in-process things, which you know are available during one specific request, will need to be available multiple times during the request but do not affect other requests.

Answer 3

In your case where you are using microservices you should use the shared cache.

The reasoning is that one of the key benefits of microservices is that you can scale them to multiple instances.

If you do this and an incomming request hits service instance 2 instead of 1 instance 2 could benefit from the cached result.of a previous call to instance 1

Unless you are using the same data multiple times within a single request-response transaction, in process caching has limited utility.