Given a set of processes running on a cluster of hosts, I'm designing a system that load balances the hosts through live migrations (carried out by an external service) of the processes.
The goal of the load balancing is to minimize or prevent resource starvation, a situation in which processes are not allocated the amount of resources they want to consume. There are several possible cases:
(1) All hosts in the cluster are overloaded, and we want to distribute resources evenly across demanding processes.
(2) We have an imbalanced cluster, we want to bring it to a balanced state as soon as possible while minimizing the cost.
I was thinking of creating endpoint in each host used for health check. The frontend is scheduled to send health check request to the host through load balancer. The frontend decides if the the host needs to be replaced or a new host needs to be added. The host location is stored in a read-heavy db.
Regarding the live migration, I believe we need either sticky sessions or if the session is offloaded to a cache service, we update the location to the new host used by frontend.
Am I on the right path? What are the core factors I need to take into account in designing this system?