3

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved in a table). If the owning node fails, the manager node can reassign it.

It is desired that once any node joins the cluster, it takes up some of the load from other members. What makes this difficult is that the entities are updated all the time by the owning nodes thus making this reassignment difficult. Any ideas on how to make this reassignment possible without interrupting the running nodes. I understand that this depends a lot on the implementation but would like to know thoughts on this.

The entity is assigned to nodes through a queue to which all nodes register as listener.

Improve this question
2
  • Do nodes pick up entities by themselves or are the entities always assigned to nodes by the manager node?
    – Dan Cornilescu
    Oct 21, 2016 at 5:20
  • The entity is picked up randomly by the nodes but this ownership of entity is registered in a table.
    – user2599052
    Oct 21, 2016 at 5:38

2 Answers 2

Reset to default
3

Personally I dislike a self-serve approach, IMHO it's much more difficult to manage reliably distribution of tasks among nodes with a distributed intelligence since the local logic doesn't have the global view of the system for its decisions. Redistributing entities when new nodes are added is just one example.

The apparent system scalability which might have been the reason for choosing such approach is just an illusion - the need for the centralized ownership registry is an indicator of that, managing it reliably in a distributed manner is no trivial task, glitches would cause jobs stuck/forgotten/lost. Error recovery is difficult, the centralized manager node is still required since the local intelligence may dissapear.

Anyways...

Since you mentioned that

once an entity is assigned to particular node, it is taken to completion through that node only

the key to allow new nodes to pick up part of the overall load is to not distribute entities to nodes until they are able to immediately start and complete their processing confortably within their load capabilities. In other words don't create local waiting queues for each node as entities may be waiting there while other nodes could easily process them immediately. And don't overload nodes just to keep the waiting times small. Keeping the entities unassigned makes them available for any node to pick up - a single waiting queue for the system.

As for balancing the load across nodes it's important to reflect the node's load in the local distribution logic. One way of achieving it without precisely knowing the loads of all the other nodes in the system is to "encode" the local node's load in the waiting queue polling rate.

For example, a node which already has 3 entities assigned would check for available entities once every 3 seconds, while a node with 10 entities assigned would poll only once every 10 seconds. This allows nodes with lighter loads to pick up entities faster, thus catching up with the other nodes - a newly added node would be able to pick up 4 entities in the same interval in which a node already processing 10 entities could only pick up one. The overall system's load would thus automatically be balanced across all available nodes (when all nodes reach the same polling rate).

Improve this answer
6
  • The setup we have is a single queue(per message type, hence multiple message types for each entity) and each node registers a poller on the queue for message type. So, I believe that poller frequency adjustment should be possible.
    – user2599052
    Oct 21, 2016 at 12:56
  • Hm, this info changes the context of the question and should be an update to it.
    – Dan Cornilescu
    Oct 22, 2016 at 18:38
  • In this context changing the polling frequency of the node may also affect the service quality for the entities already assigned to the node. Doe the node check if the entity targeted by a message is not yet assigned and, if so, assigns it to itself? If not clarify the question to indicate when and how the node picks up the entities.
    – Dan Cornilescu
    Oct 22, 2016 at 18:51
  • The service quality should not get affected because the polling frequency can be changed on that queue only through which entities come into the system. All the subsequent interactions for a entity take place through other queues which can continue to work at a normal rate.
    – user2599052
    Oct 23, 2016 at 12:24
  • Watch out for a corner case: if all nodes in the system somehow manage to all be in waiting mode at the same time and a new entity mesage comes in none of them will pick it up. Not an issue if you have some retry mechanism in place for such messages.
    – Dan Cornilescu
    Oct 24, 2016 at 11:13
1

If new tasks arrive often and are processed quickly, the easiest/best way to populate a new node in the cluster is to simply wait and let your manager node assign tasks to the least busy node. This is how load balancers in front of web servers work.

If the tasks are too big/too long for this to work well, then a simple option would be for the manager node to send a message to some of the youngest tasks to fail/exit/terminate. Your scheduling code would then reassign them to the least busy node.

If the tasks are too big/messy to do either of the above, I'd seriously think about refactoring the tasks to make them smaller rather than try to figure out how to move them from one node to another at any arbitrary point in their execution.

It's easier to change a tire when the car is stopped.

Improve this answer
4
  • The task is long(e.g - an user order)and takes multiple iterations with interfacing systems to complete. Hence, yes it is about moving them at an arbitrary point without risking simultaneous execution through old node/new node. To provide more detail, messages arrive on queues all the time and are picked by owner nodes. There should be some way to indicate them to ignore the tasks being transferred or something else. Not sure.
    – user2599052
    Oct 20, 2016 at 16:27
  • I had a feeling that was the case.
    – Dan Pichelman
    Oct 20, 2016 at 16:37
  • Do you have any suggestion on how to handle this ?
    – user2599052
    Oct 20, 2016 at 16:39
  • 1
    The long jobs have to be broken up into smaller ones with defined checkpoints or hand-off points. You can then ask a node to pause at the next checkpoint, alert the manager and the entity that things are changing and allow the manager to assign a new node to the entity.
    – BobDalgleish
    Oct 20, 2016 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.