2

I'm playing with a mental model for distributed actor system that communicates using messages. In this, it is possible for an actor to create another actor. I've encountered one specific situation where I'm not really sure how to resolve. Consider this sequence of events:

  1. Actor 1 requests creation Actor 2. As part of the "construction", it passes its own ID.
  2. Actor 1 requests its own deletion.
  3. System sends KILL message to all Actors who subscribed to {Actor1|KILL}
  4. Actor 2 is created, saves ID of "1" sent to it as construction parameter.
  5. Actor 2 attempts to send a message to Actor 1.
  6. ???
  7. Actor 2 detects that Actor 1 is gone, and reacts.

Since Actor 1 and 2 can be on physically different machines, querying "Does Actor X exist?" every time (or before every time) you attempt to send a message will introduce latency that shouldn't be there in the typical case. Similarly, blocking until you know "Send Success" or "Send Fail" seems like a bad idea. It also seems that litering code with if(send() == fail) { ... } is ugly and error-prone. Are there known solutions for robustly and cleanly handling these sorts of situations?

  • 3
    If I remember correctly TCP deals with this by sending off up to N messages at a time assuming it'll eventually get responses. As acknowledgement messages come in, TCP fires off some more. If a response is never received for a particular message, it re-requests with increasing delays (to avoid choking the network) until it just gives up. This allows you to avoid the latency of waiting for an acknowledgment on individual messages before moving on, but as long as you keep track of which messages haven't been acknowledged yet you can still detect problems. – Doval May 20 '14 at 20:12
  • The thing is, TCP wants a reply to confirm 100% of the messages. In this case, requesting a "received receipt" would be something optional. – PatrickB May 20 '14 at 20:35
  • 2
    I think what Doval was saying is you can look at how TCP does it and modify from there. TCP sends its messages with a sequence number and the receiver responds with the sequence number it received. The sender then knows exactly which messages were sent successfully or not. You can do the same thing with your messages, give them a sequence number, attach that message to something like a WaitHandle/Timer and continue processing. When a response is received close the WaitHandle/Timer with matching seqnum. If no response is received the WaitHandle/Timer will signal and you know the send failed. – Dunk May 20 '14 at 21:43
  • Many implementations of the actor model, notably Erlang, are predicated on the idea that message delivery is unreliable and that if an actor needs to know a message reached its destination, it should look for an (equally-unreliable) acknowledgement from the other end. – Blrfl May 21 '14 at 3:44
3

This problem has been faced may times when trying to build a reliable communication protocol on top of an unreliable one. That starts with TCP (as the underlying IP protocol is unreliable) and any protocol that uses UDP as transport mechanism.

The basic idea is that each message/packet gets an identification number and the recipient sends an acknowledgment of the messages/packets that it received. If the sender doesn't get an acknowledgment within a certain time frame, then the message is considered lost and the sender retries sending it. After a certain number of retries, delivery of the message is usually given up and a failure is reported to the application.

On the application level, there are two main ways to handle it:

  1. A blocking send function. This should only be used if the timeouts are relatively short. You don't want to block your actor for a minute because the recipient is difficult to reach. A blocking send function can either return a status indicating success/failure, or it can throw an exception (if failure is considered an exceptional situation and/or it is likely that the direct callers of send can't really do anything with the error except passing it on up the stack).

  2. A non-blocking send function. In this case, the failure notification will often be in the form of a callback or event that arrives some time after sending the message.

In either case, for messages where the actor isn't interested in the success/failure of delivery, the actor can just choose to ignore failure indications, or there can be a mechanism to say to the lower layers "don't bother retrying this message".

  • In a sense this is about "reliability", but it's less about guaranteeing delivery (since if you assume 0% packet loss, the same situation still holds) between active actors, it's about detecting actor destruction. It might make more sense to simply process create messages before delete messages since the problem seems to be that the program order is "CREATE 2" "DELETE 1", but the order executed is "DELETE 1" "CREATE 2". Thinking about TCP does make me think that I should sequence them... – PatrickB May 21 '14 at 14:30
  • @PatrickB: For good reliability, actor 2 should be able to handle the situation that actor 1 is suddenly gone, without notification. This could happen due to bugs or hardware crashes as well. – Bart van Ingen Schenau May 21 '14 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.