Distributed message queue, propagating queue leader/follower information

Question

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design:

I have the following components:

• FrontEnd: request validation, authentication, SSL offloading, server-side encryption, etc
• Metadata Service: keeps queue metadata information
• BackEnd Nodes: store messages. For each queue, we'll have one leader and several followers for replication
• Node Manager (for instance Zookeeper or etcd): takes care of leader/follower coordination in the BackEnd cluster

The picture shows the flow to handle a client request. The FrontEnd does the initial request processing and queries the Metadata Service to find out the leader for the given queue, and forwards the message to the corresponding leader.

My problems are with the Metadata Service. How should it fetch queue leader information? Directly querying/connecting to the Node Manager or should I introduce another component instead? Or should I get rid of the Metadata Service altogether and have the FrontEnd directly query the Node Manager? The problem with the last option is that I would still need somewhere to store queue metadata information like creation date.

Answer 1

The problem with the last option is that I would still need somewhere to store queue metadata information like creation date.

Other than you may store other fields (like timestamp) in Metadata database,

Metadata service should scale out well for reads and be highly available as every PUT and GET call for a message results in a call to the Metadata service. And it is an AP component.

Node manager (Zookeeper) is a CP component.

So it's better to implement and maintain Metadata Service (+db) and Node Manager as separate components because they have different functionalities.

Whenever a leader changes, this information is propagated to the Metadata service database. You can either apply pull (have a scheduled executor in metadata service to get data from zookeeper, and update metadata database if necessary) or push (have a client that watches the changes of zookeeper nodes and update metadata database) model.

If you are using push model, the watcher can be either in the same Metadata service or you can split the service to write and read. For the latter option, in the diagram there are MetaReadService (which is just your Metadata Service now) and MetaUpdateService (in which you process the data change in zookeeper and updating the Meta database), and your MetaUpdateService looks like

public class MetaUpdateService
{
    public boolean writeDatabase(...)
    {
       // create new record with leaders/queues info
    }
    private boolean updateDatabase(...)
    {
      // update Meta database with leaders/queues changes
    }
    public void watchLeaderUpdate(...)
    {
      try 
      {
          new Executor(hostPort, znode).run();
      } 
      catch (Exception e) 
      {
          // handle exceptions
      }
    }
}


public class Executor implements Watcher, Runnable, DataMonitor.DataMonitorListener
{
    public Executor(String hostPort, String znode) 
    {
        zookeeper = new ZooKeeper(hostPort, 3000, this);
        dataMonitor = new DataMonitor(zookeeper, znode, null, this);
    }

    public void run() 
    {
        try 
        {
            synchronized (this) 
            {
                while (!dataMonitor.dead) 
                {
                    wait();
                }
            }
        } 
        catch (InterruptedException e) 
        {
        }
    }

    public void process(WatchedEvent event) {
        dataMonitor.process(event);
    }
}

Refer to their doc for more details.