Let's say I want to design a distributed system whose only purpose is to assign each running node a unique identifier in some small range (Say, 5-100 nodes into a 10 bit identifier). When a node is added, it should get an unused identifier. We need to support more than 1024 nodes being added then removed, so of course there needs to be some system for releasing identifiers.
From first principles, what techniques would be used to implement a system like this? How would new nodes be added to the system and learn about the others? How could identifiers be released when a node crashes while guaranteeing that no two running nodes have the same identifier? Is it possible to make a system like this fault tolerant?
Is there some method by which I could test ideas I have to see if they actually work in the presence of failure etc?