We built an application that has a list of fixed intranet URLs and crawls these periodically, transforms the data, stores it in a database.
We have the requirement that the application has some kind of automatic failover mechanism. So we ordered two servers in two different data centres. Now: We can't let the two applications on each host run at the same time, that would double the load on the spidered URLs. So when the primary goes down, the secondary should up and continues spidering. "Goes down" means we don't receive data points for a period of time in the database anymore.
My first idea:
- We have an etcd cluster available, so we can put a key there which contains information which one is active/passive.
- The two instances listen to the key and on key change start/stop sending data.
- We put a simple script on e. g. a build server that is fair to assume to be high available and the script triggers the failover if we don't receive datapoints for X time by toggling the key.
That would in theory solve the issue, but is actually quite a lot of effort to implement. At the moment the application is nicely dumb and just does one thing: spidering URLs. Now I need to enhance it with listening to an etcd key, stopping/starting the threads based on if the current host is supposed to be active, ...
My previous work experience let me mostly work with systems that deal with incoming requests and rely on a load balancer in front of them and similar structures. I never had to create a failover mechanism for an application that does outgoing requests.
Is there some simple failover mechanism (simpler than the key stuff I described before) that I'm overlooking, that has minimal/no manual intervention? We also have a load balancer cluster available, but I can't see how that would help when the requests go out instead of coming in.