Managing setTimeouts and setInterval in a distributed environment

Question

I am building a turnbased game on nodejs using socket.io and redis as the datastore. I am planning on hosting the game on AWS opsworks with the ability to scale by adding more nodes.

How should I structure the application so that I can have setTimeout like behaviour to work correctly in the event that the node that triggered the setTimeout goes under for some reason?

My initial plan is to have every setTimeout get stored in redis under a sorted set, with the score of the event as the unix timestamp that it should get executed in. I see 2 major problems with this approach that I am not sure how to tackle:

1. The time of the node that reads the event may not necessarily be in sync with the time of the node that stored it, causing events to fire earlier or later than they were supposed to. I'm not sure how big a problem this might be as a quick googling reveals that the drift will be minimal at worst, but I lack any practical experience to make a judgement call on this.

2. There is no way for any of the 'worker' nodes to know that there might be a task due so they would have to poll redis every second to see if there is a task that they can handle. The ideal solution for this would be a publish/subscribe that has a delay built in but this does not seem to be straight forward to implement either.

Any advice on how to solve this is much appreciated!

Answer 1

Best soultion to this is to use a job queue built on redis. With that you can schedule a task to run at a given time. That should get rid of all need for setimeout or setinterval. In the past, I've used bull (https://github.com/OptimalBits/bull), it has served me well, but there are others.

Additionally, your initial plan could work as well. If you need to keep server times in sync, you should use ntp. As for point 2, I don't think it's a big deal to poll for tasks. Should be a simple get to see of a key exists.