Best practices for calling from one microservice to another in a loop

Question

Let's assume we have a use case where ServiceA needs to make several calls to ServiceB. I know it would be best if the calls could be consolidated as one request, but let's say that's just not possible for this use case.

My question is who should be concerned with not overloading ServiceB? Should ServiceA trust that ServiceB will have some appropriate rate-limiting and be able to deal with a surge of requests? Or should ServiceA implement some limit on its end in terms of how many requests it makes at one time to ServiceB? For example, make 3 requests to ServiceB, only once those are resolved make another 3 requests?

It seems to me that ServiceA would only be guessing at the capacity of ServiceB, and any limiting by A will not even be possible for asynchronous requests to B

Answer 1

My question is who should be concerned with not overloading ServiceB? Should ServiceA trust that ServiceB will have some appropriate rate-limiting and be able to deal with a surge of requests? Or should ServiceA implement some limit on its end in terms of how many requests it makes at one time to ServiceB?

Blind reliance is never a good thing. If you only implement one regulation, then the other service will always be stuck with blind reliance, which is never a good thing.

Firstly, the last line of defence argument applies. In other words, your B service needs to be able to protect itself from being DOS attacked. Even if you weren't intentionally sending multiple requests in quick succession, that would be a really good idea.

The specific implementation is more complex than I can pen down here based on the info in your question, but the basic implementation is that service B starts returning HTTP 429 (Too Many Requests) once the limit rate has been reached. Whether that limit is global, per source IP, per JWT; based on active requests, request per specific time period, time since the same requester sent their previous request ... is all up to you to decide.

As a last line of defence, a global request rate is a good idea, though you'll usually want to add some additional rate limits on top of that. For example, you could cap it globally at 10k requests per minute, but cap individual callers (based on IP or JWT) to 500 requests per minute. This ensures that you can fully service 20 consumers (to their full rate limit) at any time, most likely more since not everyone will be using up their rate limit.

Any consumer such as A should then notice these 429 responses and act accordingly. Depending on the context, this could mean waiting a few minutes, waiting a longer period, or simply taking more time inbetween requests. It all depends on how your rate limiting is configured.

That being said, for any non-trivial amount of requests being fired from A to B, I would already expect A to have some degree of staggering (i.e. delay timer) between firing requests. It all depends on how many requests we're talking about here. 5 requests are nothing noteworthy, 500 requests become an issue for below-enterprise-grade software, 50k requests are always going to be an issue.

Assuming you're working asynchronously, in order to receive optimal throughput you'd need to get a rough grip on how many requests your B server can handle, how many tasks your A server can fire off at the same time, and the available bandwidth between them. Depending on the bulk size we're talking about (and the average response time), this fine-grained control may be overkill, but with large bulk requests it may yield decent optimization without unintentionally DOSing the B server for any other consumers.

Answer 2

My question is who should be concerned with not overloading ServiceB? Should ServiceA trust that ServiceB will have some appropriate rate-limiting and be able to deal with a surge of requests? Or should ServiceA implement some limit on its end in terms of how many requests it makes at one time to ServiceB? For example, make 3 requests to ServiceB, only once those are resolved make another 3 requests?

Both ServiceA and ServiceB should implement strategies to avoid overloading ServiceB.

First of all, the developers of ServiceB should document the highest rate at which ServiceA (or any client) is supposed to make requests and ServiceA should honor that limitation. This documentation could tell that ServiceB is capable of handling a sustained rate of 100 requests per second (per client) and bursts of up to 500 requests in a single second, lasting no more than a second. To handle such bursts, ServiceB would probably implement some queuing internally.

Secondly, if ServiceB finds itself in a situation where it can't handle all requests (for example, when more clients than expected are making requests at the maximum rate, or a too long burst), then it is nice if it has a mechanism with which to tell clients to back-off with their requests, like returning a 429 HTTP Response code.

Answer 3

Service A should rate limit its outgoing calls to service B. Not doing so enables propagation of a cascade failure between the two services.

Under normal steady state operations where there is sufficient provision of capacity in both service A and service B, then everyone is happy, and there is nothing to worry about.

However, when a failure occurs. e.g. a background worker thread goes CPU hungry on the server hosting service B, then all of service B requests will take longer to handle. The requests being generated by service A will keep arriving at the same rate, increasing the number of open connections on both servers. Eventually the server hosting server A will reach one of its limitations. It might be server memory for all of the inflight transactions, that are now taking much longer due to service B's high latency, but it could be reaching the servers open socket limit, disk space, or that clients of service A are experiencing network timeouts.

It doesn't matter, the failure in service B has now cascaded to service A, and is on its way to being cascaded to the clients of service A.

Answer 4

Service B should return a 429 and service A should implement retry logic. Take a look at https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 for further details.

Service B can return a Retry-After header to let service A know how long to wait.