Architecting a distributed file processing system with leadership election

Question

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice:

Problem : There are a large number of files continuously posted on an FTP server that we need to grab, process and pass on.

Solution Idea: A master node will look for new files on an FTP server, and assign processing work to child nodes. The master node will send a JMS message to the child telling them which file to process and the child will send a response back when it is done, and ask for more work.

If the master node goes down for some reason, one of the child nodes should presume the role of master. My idea for implementing this was to have a "lock" collection in a MongoDB that contains info about the master node, as well as a lock expiration time. Every 15 seconds or so the master node will refresh their lock and update the expiration time to 30 seconds in the future. If the child nodes sees the lock is expired, one of them will assign itself as the master node.

I'm looking for feedback on this design, and wondering if anyone has advice on improvements/java frameworks or tools that already exist that I can leverage for something like this.

Thanks!

Answer 1

Well, the "leader election" problem is quite well known and the most commonly used app for solving it is probably Apache Zookeeper.

Google it and you'll find plenty of documents about that.

If you want a real-world example - this is how Apache Hadoop is implementing HA using Zookeeper - https://github.com/hopshadoop/hops/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java

Answer 2

Can the worker nodes see the filesystem where the files arrive?

If yes, do not use any other channel to communicate with them - instead do all the signalling via the filesystem using standard old-school Unix techniques and the semi-atomicity of low-level Unix FS operations.

If no, then how will the worker collect the file for processing? E.g. Are the files too big to be sent over JMS directly?

As the design stands, you have 3 independent pathways:

• ⁠Asynch notification mechanism (JMS) • ⁠Data delivery (??? - how do the workers actually get the file bytes to process) • ⁠Distributed locking (currently proposed as a database)

This is too many. These types of solutions can be done with only 1 mechanism (in the case where the workers see the filesystem directly, or can scan it) and with 2 (the case where the workers don't have direct access to the FS).