Say there are N nodes/pods (which can change dynamically), and there are a list of X data sources.
We need to distribute the responsibility of these X data sources evenly across these N nodes, so that each node knows which data source they have to scrape data from?
What would be a good way to do this?
What i have tried :-
Use another service that does a heartbeat to these N nodes and distributes the X data sources. But this new service would be a Single point of failure.
Consensus - too complex