-4

Say there are N nodes/pods (which can change dynamically), and there are a list of X data sources.

We need to distribute the responsibility of these X data sources evenly across these N nodes, so that each node knows which data source they have to scrape data from?

What would be a good way to do this?

What i have tried :-

  1. Use another service that does a heartbeat to these N nodes and distributes the X data sources. But this new service would be a Single point of failure.

  2. Consensus - too complex

Improve this question
3
  • Sharing your research helps everyone. Tell us what you've tried and why it didn't meet your needs. This demonstrates that you've taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask – gnat Mar 23 at 11:51
  • You seem to have ruled out the two primary methods of distributing work across nodes, so I'm not sure what kind of assistance you want this community to provide. My $0.02: use an orchestrator service and make it robust so it isn't a single point of failure. – Dan Wilson Mar 23 at 12:52
  • Look up Distributed Hashtables and Amazon's Dynamo system (not DynamoDB). – user253751 Mar 23 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.