1

I have a question about best practices given the following scenario.

Requirements

There is a daily process that has two steps:

  1. Query configuration data across dozens of database shards
  2. Perform many intensive jobs based on that configuration data, across a cluster of applications

The first step is lightweight and can only be performed by a single application. Otherwise, duplicate queries will get run. Afterward, the jobs can be distributed across the application cluster.

Question

How to enforce that only one application in the cluster performs the first step?

Ideas

  1. Database locking
    • Each application starts up at the same time (e.g., via cron) and attempts to get a lock. Only one will ever succeed. After this winner performs step #1, it distributes the work across all applications in cluster.
  2. Submit single queue message to cluster each day, whichever application polls it first will perform the queries in step #1 and then distribute work across cluster (including to itself).
  3. Create a separate, singleton application (outside of the cluster) to perform step #1 and then distribute work.
1

It seems to me like your two steps are largely unrelated. While there is a data flow dependency between them, you are already planning to perform them separately. So make them entirely separate processes that communicate through a message queue.

This is basically the middle ground between solution 2 and 3. You can still schedule the configuration gathering job on the cluster. Deploying all your processes uniformly will likely be easier in the long run. Since this configuration job will be idle most of the time, it doesn't need a dedicated cluster node and will consume negligible resources.

A direct implementation of your solution 2 may not be a good idea depending on how your message queues are configured. 1:n or n:1 queues offer much simpler data flow than n:m queues that are needed for an “every process could write a message, and every process should receive a message” architecture.

  • by deploying uniformly do you mean (let's say) one configuration gathering job and N "working" jobs by cluster node? – Laiv Feb 26 '18 at 19:35
  • @Laiv By “deploying uniformly” that all processes (config + worker) should be deployed via the same mechanism: as jobs on the cluster. Not some jobs on the cluster and others externally. I didn't expand on this in the answer, but since these processes are connected they should generally be developed and deployed together. Putting both in the cluster eliminates unnecessary differences. A process with only a single instance is still a valid job for the cluster. – amon Feb 26 '18 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.