Sharing akka actors on multiple backend instances

Question

I have a web applications (angular) which is connected to backend using websockets. The role of web app is to display data from multiple sources (which are updated constantly by some Scala Spark apps) in dashboards;

because the backend is deployed on multiple instances (3x centOs), the frontend will send the request to a proxy, which will redirect on one of the backend instances.

On the backend side, the connection with frontend is made using Akka Actors. Each client actor (which is equivalent as each browser client) have a request body which must be saved in memory (dashboardId which is displayed on page, a list of widgets and some data). So, for each new connection, I save in a list: actorName + this request body. To centralize the data from all backend instances, I save this list on a Redis instance.

The problem: If a 3rd party app will push a request to my backend through the proxy (no matter which instance), it will trigger an event inside the backend: the own backend will create an client and will go on a case which will push fresh data for all listeners (client actors). Because this event is isolated (and physical the actors on same instance) on a backend instance (let's say first: backend1), the rest of clients will remain out of date, even if from Redis I take all actorNames which must be updated (but ~ 66% of them are created on another servers);

controller:

 val server = system.actorOf(Props[ServerActor], "ServerActor")
 def ws: WebSocket = WebSocket.acceptOrResult[JsValue, JsValue] { request =>
   ActorFlow.actorRef { out =>
    Props(new ClientActor(out, server))
   }
 }

server:

class ServerActor extends Actor {
  def receive = process(Set.empty)

  def process(subscribers: Set[ActorRef]): Receive = {
    case Join(ref) => context become process(subscribers + ref)
    case Leave(ref) => context become process(subscribers - ref)
    case Push(request) => {
       // get all subscribers from Redis 
       // which have same request body as the new event
       val subscribersFromRedis = getActorsByRequest(request)
       val subscriberNames = subscribersFromRedis.map(_.actorName)

       // get client actors from THIS instance, 
       // because they physical exists 
       val subscribersFromMemory = (subscribers - sender())
            .filter(s => subscriberNames.contains(s.toString))
            .map(s => (s, subscribersFromRedis
              .find(x => x.actorName == s.toString())
                .get.options)
            )

       // anyway, here will remain a list of actors 
       // which physically exists on another backend instances
       val subscribersFromAnotherInstances = subscribersFromRedis
         .filter(x => !subscriberNames.contains(x.actorName))

       // send data for subscribersFromMemory
    }
  }
}

client:

class ClientActor(out: ActorRef, server: ActorRef) extends Actor {
  server ! ServerActor.Join(self)

  def receive = {
     case json: JsValue => // send data
  }
  
  def postStop = {
    RedisClients.delete(self.toString) // remove actorName from Redis
    self ! PoisonPill
  }
}

I heard about Cluster sharding, read some articles because had a similar problem in past, but can't understand how can be this solved. In my mind, I understand that I need put the client actors in another level than backend instances, somewhere centralized, behind the backend to be possible that each of it to access entire list of actors.

Answer 1

I would take a little different approach. Instead of saving the data in Redis I would use Redis Pub/Sub to share the data incoming request between the different backendes.

For example:

Backend 1:

• Client 1
• Client 2

Backend 2:

• Client 3
• Client 4

Backend 3:

• Client 5
• Client 6

So each backend has a actor for the respective client.

All three backendes subscribe to a Redis queue "app1".

Now let´s assume that 3rd party app1 sends information for client 1,2 and 6 to backend 3.

Backend 3 publish a new event to Redis queue "app1".

So all three backendes become the event that new information for client 1,2 and 6 are available.

• Backend 1 will go through all actors and see that is has to send the information to client 1 and client 2.
• Backend 2 will go through all actors and see that it has nothing to do and simple discard the events.
• Backend 3 will go through all actors and see that it has to send the information to client 6.

So client 1,2 and 6 get the information to update the dashboard. And each backend has only a list of the actors and the corresponding clients which are connected to this backend and don´t even know how many other backendes exist.

I hope this is helpful?