Shared access tokens in a distributed system

Question

I'm building a system that horizontally scales with many nodes. Any of these nodes could consume an API that's hosted on https://example.com/api. The authentication for this API is token based.

My question is, should each of my nodes authenticate separately and fetch their own token? Or should I build a shared cache between them and store the token there?

If latter, are there any standards for this design pattern, or better yet libraries that can do this for me? ie. stores the token in cache, and takes a lambda function + ttl for auth or renewal)

Notes:

• Tokens are valid for a certain number of hours.
• They're reusable during that time period, and then expire.
• They're all the same access level.
• The APIs that I've encountered so far, all allow for multiple active tokens at the same time, ie. if you get a new one, doesn't invalidate the old ones. (But I think I should account for this in case I see one in the future, not mvp)
• The nodes do run on different machines.
• The concern is not to have to make a separate Auth call on every API call

EDIT I've also ran into this article that talks about this problem in Azure world: https://docs.microsoft.com/en-us/azure/architecture/multitenant-identity/token-cache

Thanks!

Answer 1

The simplest solution, I believe, is to use one token per node:

1. You don't have to use a shared cache. The difficulty here is to handle the case of two nodes brought online at the same time: if the first node starts requesting a token, the second one should wait, but it also means that the second one should be informed that the token is being requested. While technically feasible, this is not the easiest thing to develop and to test.

2. You don't have to think what happens when the token expires. Imagine you have ten nodes. The token is about to expire. You should possibly elect one of the nodes to renew the token. What if, once elected, the node starts requesting the new token, but crashes before it saves it to the store? How other nodes would be informed about that? Would they perform a new election? What if, while the nodes continue crashing, the token finally expires?

3. Once the token is renewed, how does every node know that the new token is available (unless they query the shared cache every time)? What if a node cannot access temporarily the shared cache?

4. Imagine you are alerted that the number of requests to the API were multiplied by one hundred for the past fifteen minutes, and that you're quickly reaching the quota. If you can pinpoint specific nodes as the culprits, you may easily find the reason, such as a buggy release of the app which was deployed on those nodes specifically, or a security breach with those specific nodes being compromised. If you don't have this information, well, it's one piece of information less you can rely on during debugging.