4

I am trying to understand the advantages of service discovery compared to load-balancing within the context of a microservice mesh where many instances/nodes/VMs/containers of many web services are all calling each other in a complicated web-pattern. And my understanding of the situation is this:

  • Service discovery -- when implemented properly -- keeps each instance of each service aware of all the valid IPs/addresses of all the services it cares about
  • Each service can then employ its own client-side load-balancing to determine which instances of which services it makes a request to
  • Traditional service-level load-balancers have the problem where an instance sitting behind them becomes unhealthy or goes offline, and they continue serving traffic to it

Hence let's say we have Microservice A and B, and A needs to make requests to B. In a traditional load balanced setting, A and B would be behind their own load balancers respectively, say, https://a.example.com and https://b.example.com. When any instance of A need to make a request to B, it has https://b.example.com in its configs as the host to make contact with, and the load balancer there will farm the request out to whatever healthy nodes sit behind it.

The problem here, again, is that a B instance might have gone offline since the last health check to it (from the B balancer) and now that B balancer unknowingly sends a request off to it.

Also, the load balancer still needs to know which instances to route traffic to, which is a configuration that typically needs to be done manually. The solution to this, in my mind, is have automated registration/de-registration of instances with their load balancer.

So properly-implemented service discovery swoops in and takes care of this scenario. The service discovery tool (Consul, etc.) is constantly pinging each instance of both service A and B and letting each instance of A know the hosts/IPs/etc. of each healthy instance of B. But now its up to A to come up with its own client-side balancing solution for which healthy instance of B it connects to.

So to begin with, if anything I've said above is incorrect or misled, please begin by correcting me! Assuming I'm more or less understanding the lay of the land here, I'll move on to my question:

My question

To me, this just seems like a timing issue with balancer health checks. If the balancer can be written/configured to check the health of the instances sitting behind it just as regularly as the service discovery (Consul, etc.) tool would, and if the instances are always registering themselves with their load balancer at startup time, then the balancer will have the same information available to it as you would have in a service discovery paradigm. No?

Quasi-believable, pseudo-real world example to work with

  • Say I have an Order Service (orderws) that has a "order item" endpoint exposed at, say, POST /v1/orders
  • Let's say all the orderws nodes are sitting behind a load balancer at https://orderws.example.com
  • Say this orderws calls 2 other web services:
    • a Payment WS to make payments and transfer money, paymentws exposing a POST /v1/payments endpoint with all nodes behind load balancer at https://payments.example.com; and
    • a Shipping WS (shippingws) that, upon successful payment of an order, results with the ordered item being taken off the shelf at a warehouse and shipping to the customer; the Orders WS calls POST /v1/shipment endpoint with all Shipping WS nodes behind https://shippingws.example.com
  • Now then, how would service discovery and load balancing compliment each other here? Ideally, if even 1 Payment or Shipping node exists, their load balancer will serve traffic from the load balanced URLs. If no nodes exist behind a balancer, then what would happen (service discovery-wise, between Order WS instances and the Service Registry) with Order WS nodes attempting to handle order requests?
7
  • 1
    If all service B instances are behind a load balancer, then why would service A need to be aware of them, and why should service A care if one of those instances happens to go down? That seems to defeat the point of having the load balancer in the first place. Aug 5 at 15:58
  • Hi @BenCottrell (+1) well again I'm asking this question from the lens of someone who thinks they understand service discovery, but I'm not 100% confident of that. This is why in my question I first asking for someone to provide correction if my base understanding is flawed. But assuming its not, then my question answers your question (I think). The load balancer could be serving traffic to an unhealthy node which doesn't doe the A-client instance any good. Aug 5 at 16:25
  • I believe this scenario can't happen in a service discovery paradigm because the discovery server notifies A instances immediately when that unhealthy B instance reports as being unhealthy. Keep me honest though, maybe I'm not seeing the forest through the trees here! Aug 5 at 16:25
  • 2
    Service discovery can't prevent that happening, and indeed just because a process is reporting healthy doesn't mean that requests will succeed - there are many things which could potentially go wrong that a healthcheck might not be able to detect. For example, the instance could have been healthy at the moment it received the request, but then it might crash halfway through processing -- in which case, service A sent its request to a healthy service, but never gets its response. The bottom line is that service A must always be able to cope with its requests failing no matter what. Aug 5 at 17:42
  • 2
    However, the purpose of service discovery isn't about reliability, it's about reducing the burden of configuration management -- i.e. a feature which means that a human doesn't need to keep manually editing configuration data by-hand whenever the availability of a particular service or instance changes (for example, services running on servers with dynamically allocated IP addresses, or which use dynamically allocated ports would mean the 'route' to a service is likely to change frequently - horrible if a human has to deal with by-hand, but carefree if service discovery takes care of it) Aug 5 at 17:45

1 Answer 1

4

There seems to be a misconception about both service discovery and load balancing. There is no "versus" comparison, because service discovery and load balancing are independent, but complimentary concepts.

Load balancing a service allows clients to be decoupled from the scalability of those other services. All clients have a single URL to interact with. Cloud environments have automated tools that can add and remove nodes behind a load balancer. This helps enable the scalability promised with micro services. Service discovery has nothing to do with this aspect. Literally, this is why you would put a load balancer in front of a cluster of nodes to scale a micro service.

If load balancing decouples clients from the scalability of a service, then service discovery decouples clients from knowing which URLs can be used to communicate with the other services. Think of service discovery as an index of all the micro services in your ecosystem. The meta data about each service should return the URL of the load balancer in front of a service.

Load balancers allow you to add and remove nodes for a single service without affecting clients. Service discovery allows you to add and remove entire services without deploying config changes to the entire ecosystem. Clients can discover the new endpoints on their own.

So, don't choose one over the other. Choose both. They work well together, because they serve different purposes.


Note that resiliency in micro services is not fully achieved at the load balancer level. Concepts like eventual consistency, and utilizing simple message queues helps make the entire system resilient. The system should be resilient from the perspective of entire services, not individual nodes. This is what Martin Fowler calls "dumb pipes (message brokers) and smart endpoints (the service reading from the message queue)." The simpler a component is, the less likely it is to fail.

Message queues should be pretty simple, so they do not fail often. The service is complex and more prone to failure. If the service goes down, then messages pile up in the queue. When the service comes back online, then it starts processing the backlog of messages. This is the heart of resiliency in micro services.

8
  • Amazing answer @Greg (+1 and thank you). I'm following the theory of what you're saying, but failing to see the forest through the trees, so to speak. I updated my question with a quasi-realistic example. Any chance you would be willing to take a look at it and address the question I raise from that example? Essentially, how are entire load-balanced services (served from behind a load-balanced URL) moved in and out dynamically such that: (1) if there are 0 nodes behind a balancer, it entire service is out of order... Aug 5 at 23:17
  • ...and (2) anything trying to use the downed service knows not to try and call it? Aug 5 at 23:17
  • 1
    @hotmeatballsoup: Service discovery doesn't help when services go down. An unexpected failure is not the problem service discovery solves. Unregistering a service is done as a planned action. I don't see the point of a service discovery app doing anything if a service experiences an outage. If you try to call my mobile phone, and the line is busy, would you delete my contact from your contacts list? Same idea with service discovery and outages. Aug 6 at 0:56
  • 1
    In reply to your example, this illustrates a common problem with traditional service-oriented architecture. A cascade of service calls can fail simply because one service stops responding. There is no fault tolerance. At that point, neither load balancers nor service discovery can help you. You need dumb pipes and smart endpoints to recover from that — think "message queues" and "eventual consistency". Something needs to retain requests while the service is not responding, or the whole house of cards comes tumbling down. Aug 6 at 1:01
  • One last followup question, if you don't mind @Greg -- and I sincerely thank you for your considerate answers up to this point: in your opinion, what should the granularity of an API Gateway be? Should there be a 1:1 correlation between Gateways and Load Balancers (per service)? I ask because most OSS gateways (Zuul, Kong, etc.) also provided load balancing as a feature, and taking everything you're saying here into consideration, I don't see a way to decouple balancers/services from gateways if they gateways + balancers are packaged together... Aug 6 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.