0

I am developing a distribution id project, now using the twitter snowflake id as the fundation of the distribution id. In kubernetes cluster, to fetch the uniq and non-conflict datacenter id and worker id is hard. Now I am using the podip % 32, as the pod increase, the worker id conflict increase.

I have an idea to using kubernetes statefulset name and define the pod name and worker id mapping relation ship as const map, because each pod of kubernetes statefulset ip and name are fixed. For example, I define the mapping relation ship like this:

service-1  datacenter id is 1 and worker id is 1

service-2  datacenter id is 1 and worker id is 2

service-1  datacenter id is 1 and worker id is 3

when start the pod, the worker id will never conflict when the pod less than 1024. is that a good idea? if I using redis or zookeeper, that make the system relay on third part system and more complex. any suggestion is welcome.

2
  • have you thought of using string ids rather than ints?
    – Ewan
    Commented Jun 23 at 11:02
  • I have considered using uuid but the ints have a good performance when query and save more space. @Ewan ints was the best choice righ now.
    – Dolphin
    Commented Jun 23 at 11:59

1 Answer 1

2

Using the ip address to generate a worker id is flawed. For example a pod can have more than one container.

You should separate your seed number generation from the underlying infrastructure completely.

  1. You should deploy a datacenter id service, ensuring that there is only one instance per datacenter.

  2. In its config you assign the datacenter service its datacenter id

  3. When you deploy an instance of a id generating service you should set in its config which datacenter id service it should use to get its worker id.

  4. When the id generating service starts up, it should generate a UUID in memory and send this to it's configured datacenter ID service asking for a datacenter ID and worker ID.

  5. The datacenter ID service should reply with its datacenter ID and generate a new unique worker ID, which it should assign to the UUID sent in the request.

  6. You have to deal with the problem of running out of worker IDs in your limited pool. The workers can save their UUIDs to disk so they can reuse them, or have a leasing system for the IDs where they only use them for a set period of time before re-requesting etc.

This setup allows you to spin up the maximum number of workers while guarenting no duplicate worker IDs regardless of whether you are using containers, VMs, physical machines or multiple processes etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.