I have a question about best practices given the following scenario.
There is a daily process that has two steps:
- Query configuration data across dozens of database shards
- Perform many intensive jobs based on that configuration data, across a cluster of applications
The first step is lightweight and can only be performed by a single application. Otherwise, duplicate queries will get run. Afterward, the jobs can be distributed across the application cluster.
How to enforce that only one application in the cluster performs the first step?
- Database locking
- Each application starts up at the same time (e.g., via cron) and attempts to get a lock. Only one will ever succeed. After this winner performs step #1, it distributes the work across all applications in cluster.
- Submit single queue message to cluster each day, whichever application polls it first will perform the queries in step #1 and then distribute work across cluster (including to itself).
- Create a separate, singleton application (outside of the cluster) to perform step #1 and then distribute work.