4

I have an existing Scala application that I am trying to refactor in order to use Akka. One of the problems I have is how to manage error-checking in actor-based applications.

Usually error-checking is done through one of two mechanisms:

  • either one returns a value indicating an error condition, such as Option[A] or Failure[A], or
  • by the use of exceptions.

Neither of these style seems particulary useful here. On the one hand, actor messages are usually "fire and forget", hence there are no return values. [One can have a return value using Futures, but it certainly not customary to ask for futures on every message.] On the other hand, processing of the message usually happens on another thread, so that one cannot catch an exception arising from the processing of a message.

One could simulate the first mechanism by sending back error or confirmation messages, such as

class FooActor extends Actor {
  def receive = {
    case Foo => ...
      if (errorCondition) sender ! ErrorMessage
  }
}

But if one has to do this for every actor, it becomes a lot of boilerplate and it seems a poor man's simulation of stack unwinding.

What is a good strategy to recover from errors in actor-based applications?

|improve this question
  • Check out the Akka docs. Specifically the ask pattern under the header: "Ask: Send-And-Receive-Future". You use futures and handle exceptions in the onFailure callback of the sender. This is all done without blocking. – Mike L. Jan 31 '13 at 14:37
  • Yes, this is what I added parenthetically "One can have a return value using Futures, but it certainly not customary to ask for futures on every message.". But this requires generating a Future for every call, which basically means doing away with ! and getting a fixed overhead. I do not think this is the customary way to build an actor-based application. – Andrea Jan 31 '13 at 15:27
  • Moreover, one has to send back the failure explicitly, as the docs say: "To complete the future with an exception you need send a Failure message to the sender. This is not done automatically when an actor throws an exception while processing a message." – Andrea Jan 31 '13 at 15:31
1

Based on your example with the database connection failing, I think you need to introduce another actor. This actor holds a queue of the messages that contain the data that needs to be written to the database. You then have the actor that holds the database connection request the next message to be saved to the database by sending a message to the queue actor. Then, a message containing the data to be saved is sent to the database actor and upon successful save, the database actor sends a message to the queue actor signifying a successful save. At this point, the message that was just saved can be removed from the queue and the next message sent to the database actor, if there is one. However, if the database actor crashes, the message signifying the successful save is never sent, and thus the message is still in the queue actor. When the database actor starts back up, it will simply tell the queue actor it is ready to process again, and the previous message that was not saved to the database will be sent again.

|improve this answer
  • Yes, I think that some sort of queue, possibly with a durable mailbox, is needed in front of the actors that are most prone to crashing - think actors connected to db, web services and so on. In your design, you would have a single actor who has the role of saving entities. Have you tried that? Does it become a bottle neck? – Andrea Feb 7 '13 at 0:05
  • What I have seen done is to create several database actors and one queue actor. The database actor sends itself to the queue actor when it is ready to write. The queue actor handles the pairing of messages to database actors as the messages become available. Just like any concurrent application, performance it's going to depend on your application. You need to find a balance between the number of actors that achieves the desired throughput. This may require creating multiple queue actors that each manage their own set of database actors. – Mike L. Feb 7 '13 at 18:18
2

Fault tolerance is one of the key concepts of Akka and "let it crash" is one of their mottos. Akka's mechanism for handling failures is called supervision, and it's right there on akka.io/#supervision:

Actors form a tree with actors being parents to the actors they've created. As a parent, the actor is responsible for handling its children’s failures (so-called supervision), forming a chain of responsibility, all the way to the top. When an actor crashes, its parent can either restart or stop it, or escalate the failure up the hierarchy of actors. This enables a clean set of semantics for managing failures in a concurrent, distributed system and allows for writing highly fault-tolerant systems that self-heal. Learn more.

Hence, you don't need to explicitly fire failure messages in case of an exception. The supervisor strategy of an actor's parent (or the parent's parent and so on) is invoked automatically. Although you can of course do it if you want to.

Also take a look at Supervision, Lifecycle Monitoring and Restart Hooks.

|improve this answer
  • Well, this is enough to prevent the whole system from crashing, but it is not enough to handle errors, as far as I understand. Say I receive some external input: this generates various messages, each of which at the end is sent to an actor that will save it into a database. If the database connection goes down, some of these actors will crash. Supervising is enough to bring back the actor in a working state, but if I understand correctly the message will be lost. This may lead to a situation where some messages are saved and some are not, and thus to inconsistent data. – Andrea Jan 31 '13 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.