I've been working on a large distributed system typically with over 10,000 clients living on a single network (LAN) all accessing a central set of API's in the cloud. Not exactly a challenging problem. Where it gets interesting is that each client may be requested to upload a large amount of data anything from a 1 KiB file to a multi GiB file. A single request to upload will happen ~simultaneously across the clients.
Parts of the system that are stochastic:
- Which clients are alive.
- How many clients a file is requested from.
- The size of the file.
- The number of requests for files.
My question is, are there any good white papers or known algorithms that could be used to coordinate all the clients so that they don't immediately overwhelm the network that they are residing and use up all the bandwidth? The main issue would be the edge router since they have fixed bandwidth. The expected result would be that the clients figure out a way to coordinate with themselves to eventually complete the file request while keeping bandwidth use to a minimum and that they minimize communication with the API during this process.
To date, I am just allowing a fixed number of clients to transmit large amounts of data at once. Not optimal though it works. Standard rate limiting is out as the system cannot just shut down client connections as they are integral to the functionality of the system.
My guess would be that someone working with map-reduce would have run into a similar issue. I have also, look into the possibility of creating a protocol for local and global voting on priority, similar to MongoDB election process.
You have a 100 employee company with a 300x300 backbone to the internet. They have clients installed on all their laptops and servers. Let's imagine each employee has one laptop and there are 10 servers.
If not hard to conceive a situation where there are many files spanning the majority of machines and they have been requested at the same time. This could easily cause a situation where all the bandwidth is being consumed for a brief period of time affecting delivering said files, the is likely to affect the employee's ability to do their jobs.