I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming data for import into a database.
For each customer, there may be any number of socket connections required to consume different data feeds. Creating a new ECS service for each customer is not practical as this would also require new ECS task definitions with slightly different configurations, which ultimately would result in having 1000s of services/task definitions. This approach would quickly become a maintenance and monitoring nightmare, so I am looking for a simpler solution.
The list of "feeds" is relatively static and is stored in a document database. Feeds are only added as new customers sign up. My initial thought is to have a fixed number of containers responsible for fetching the feed configurations from the database. Each container would attempt to acquire a lease for the feed, and if acquired, start the feed and let it run until the container is killed or the connection is interrupted. Each container would have to periodically check for new feeds that are available in the pool. They would also have to extend the lease while the feed is running so the same feed isn't pulled by another container. There could be a race condition here where the lease expires before it is extended, so I'd have to be careful to always extend the lease.
This solution would work, but there are some obvious sticking points. If each container starts at relatively the same time, there needs to be a way to control how many jobs each container is allowed to start at one time so that you don't have 1 or 2 containers starting all of the jobs at once. One approach would be to pull one job every couple seconds until the pool is empty and all feeds are leased. This would lead to potentially uneven job distribution, and the ramp up time might take a while until all jobs are pulled from the pool and leased. I could also have the containers fetch a feed and start it, then go grab another one. Some containers may start their feeds faster and go fetch another job before another container could finish starting its feed, thereby leading to container hot spots.
Another approach could be using something like consistent hashing. If each container can know the ID of itself and the other containers, it can hash the feed configuration ID and figure out which container it belongs on. This would distribute the jobs more evenly, but I would also have to deal with checking periodically for new feeds, for example, if a container were killed and the feeds leases expired.
I have also thought that actor programming like with Akka might be exactly for this problem, but that does not come without significant complexity in implementation.
I am open to any and all suggestions or any other holes you can poke into my existing proposals!