Fail over for application that does outgoing requests

Question

We built an application that has a list of fixed intranet URLs and crawls these periodically, transforms the data, stores it in a database.

We have the requirement that the application has some kind of automatic failover mechanism. So we ordered two servers in two different data centres. Now: We can't let the two applications on each host run at the same time, that would double the load on the spidered URLs. So when the primary goes down, the secondary should up and continues spidering. "Goes down" means we don't receive data points for a period of time in the database anymore.

My first idea:

• We have an etcd cluster available, so we can put a key there which contains information which one is active/passive.
• The two instances listen to the key and on key change start/stop sending data.
• We put a simple script on e. g. a build server that is fair to assume to be high available and the script triggers the failover if we don't receive datapoints for X time by toggling the key.

That would in theory solve the issue, but is actually quite a lot of effort to implement. At the moment the application is nicely dumb and just does one thing: spidering URLs. Now I need to enhance it with listening to an etcd key, stopping/starting the threads based on if the current host is supposed to be active, ...

My previous work experience let me mostly work with systems that deal with incoming requests and rely on a load balancer in front of them and similar structures. I never had to create a failover mechanism for an application that does outgoing requests.

Is there some simple failover mechanism (simpler than the key stuff I described before) that I'm overlooking, that has minimal/no manual intervention? We also have a load balancer cluster available, but I can't see how that would help when the requests go out instead of coming in.

Answer 1

This is a little bit obvious .... but in the world of telemetry servers, this type of primary/backup is often done by having the servers exchange "heartbeat" messages.

In this way, both servers know the status of the other at all times and can take appropriate action ... such as failover if primary goes missing or alerts if secondary gone AWAL or the app has stopped.

Often one server is designated the default prime, and when it recovers, it may force a fail-back to re-establish itself as prime. However, this may be of no concern in your application, but there can be reasons like the backup is a less well configured machine to save costs.

Note the heartbeats can be at various levels, eg simply ping to see if the other is alive and on the network, or app-to-app messages to ensure the apps or specific parts of them are running.

Obviously the ping level stuff can be done with scripts, but app-to-app may require a few extra lines of code in the apps.

Also, your app sounds like its stateless, just trawling and storing, and re-trawling the same site after failover does not matter. If your app had a lot of state, then failover can be more complicated, as you would need to take this into account in the failover.

Lastly, the above is suitable (and well proven) in warm standby situations, which again looks like what you are after.