Context: I own a service that's distributed across VMs, that themselves are distributed across multiple data centers.
Scenario: I'm adding an API that calls a single downstream dependency. This dependency has a very small throughput (relative to the API). Further, instead of just returning 429 if it's called too often, the dependency extends its retry delay with each 429. Thus, I'm concerned about a self-DOS if I don't rate-limit myself.
Design: My API will queue all requests to provide load-leveling. Right now, any VM could dequeue a request and start working on it. The API is designed in an asynchronous manner; responses are returned via a callback.
Problem: While I can rate-limit on a per-VM basis, I recognize that with horizontal scaling, I may actually make N times the allowable calls.
Question: Are there design patterns or technologies that let me restrict the scalability of my distributed system so I can rate-limit my calls?