Should I put API gateway behind load balancer or vice versa

Question

I am currently learning system design. I wonder if I should put load balancer in front of multiple instances of API gateway or put an API gateway in front of single load balancer. which one is valid design?

Answer 1

Everything needs to be load balanced and have failover.

What this means is that usually you have some shared IP address at the very top to handle your top level load balancer/api gateway failing.

Now if you are using a managed API gateway, you might get that all built in as part of the service and hidden from you.

Behind your gateway it would be normal for each service to have its own mini load balancer cluster (ie two LBs with a shared IP). This allows clever rolling/canary deployments with a controlled blast radius of just that service.

You could run it all from one load balancer cluster. But if you do that every time you change the config, to take a box out of load, or route traffic by version or something, you run the risk of taking everything down if you make a mistake.

Answer 2

It would depend on the scenario and what the load/traffic is. Consider a case where one had 2 API Gateways, one would have a load balancer in front so that each gateway gets a share of the traffic. There also might be a load balancer(s) in back of each gateway to split the load across sets of servers for a particular service being invoked.

Now consider the case on 1 API gateway. All requests come into the gateway, but then get routed to different sets of servers depending on the service invoked. Each set of servers may/will have a load balancer in front of them to balance the load across the set of servers for that service.

Most websites are not serving up millions of requests per second, so 1 gateway may be suitable. Larger traffic patterns may require multiple gateways and load balancers to handle/distribute the load.

Also, note that load balancers and API gateways serve two purposes. Typically load balancers distribute network traffic. An API gateway is typically a centralized entry point for your service layer and usually handles things like authentication/routing, logging, rate limiting, among other things.

Answer 3

I think you are confusing items, by looking at your diagram and then deciding it represents something different that it does.

First let's state a few items:

• The LoadBalancer will expose the API of whatever is behind it.
• The API is provided by a service.
• When we take these items in combination, a load balancer exposes the API of whatever is behind it.

This means you aren't asking if your API should be in front of or behind your load balancer, you're asking if you need a load balancer to balance API service calls or the instances that the API service calls then query.

The answer is, it depends. You might even need load balancers in front of and behind your API. To answer such questions, let's ask a few questions of our own.

• How will the API service scale? When it is overloaded, will you launch another one? Do you intend to reconfigure all the services that use the API to now use both, or do you want one connection point in the client configuration?

If you want to scale you API service, and you want to avoid re-configuring your clients, create a load balancer in front of the API service and configure all your clients to use it.

If you want to not scale your API service, and you want your clients to be reconfigured when you change the API service setup, configure your clients to connect directly to the API.

For your "instance A and B" connections, again you need to follow this pattern of asking questions. Sometimes it's better to balance across a large number of instances, sometimes it isn't. Balancing provides lots of flexibility, but it also adds costs. Configuration and deployment can be more complex, and it doesn't make sense when scalability of a service is more "each instance of A gets one private instance of B" instead of "each instance of A gets access to a pool of instances of B".