How to avoid points of critical failure?

Question

Background

We have a ton of GPS devices in vehicles. These devices need to communicate with our system. To achieve this, we need to parse the device's messages before proceeding.

The following describes my scalable and fail resistant architecture idea.

Architecture

The devices are black boxes that send us buffers of data via TCP every X seconds. A typical message follows this trajectory:

1. Each device communicates with a RabbitMQ server (R1) that has a persistent queue. This way if something fails, no messages are lost.

2. R1 sends the messages to the Message Parser Cluster (Hamster Cluster). Fancy name for a bunch of hamsters that receive a data buffer as input and output a JSON object we can understand.

3. The Hamster Cluster then sends the parsed messages to another Persistent Queue (R2), which has the same properties as R1.

4. R2 then sends these messages to the Data Processing Cluster (Monkey Cluster) which does the real work. We are inside our system now. The path ends.

Objectives

The main objectives here are to have an architecture design with the following properties:

1. Scalable. We must be able to recruit more Hamsters and Monkeys if any of our Clusters is dying ( No one likes dead animals! )

2. Fail proof. If any given machine in the system fails, the service must not go down.

Problems/Questions

1. As designed, this system has 2 critical failure points: R1 and R2. Should this machines fail, the whole system goes down. Is there a way to avoid having these two critical failure points?

2. My coworkers made the case for NGINX, which also supports TCP/UDP connections. I understand that using it, we would have load balancing between the machines of each cluster. However, Replacing the Rabbit servers for NGINX servers would still have 2 points of critical failure. Could this be avoided?

Answer 1

Since you are already using RabbitMq. Consider that RabbitMq can be made highly available. RabbitMq can also be made to be pretty durable.

From the RabbitMq team themselves...

For high availability we recommend clustering multiple RabbitMQ brokers within a single data center or cloud region and the use of replicated queues. Clustering requires highly reliable, low latency links between brokers. Clustering over a WAN is highly discouraged.

Disaster recovery typically requires asynchronous replication of data between data centers. RabbitMQ does not currently support this feature but other message routing features can be leveraged to offer a partial solution.

RabbitMQ does have support for replication of schema (the exchanges, queues, bindings, users, permissions, policies etc) which allows for a secondary cluster in a different data center to be an empty mirror of the primary cluster. A fail-over using only schema synchronisation would provide availability but with a data loss window.

You will have to decide for yourself if the durability (disaster recovery, or the ability to not lose data, or minimize the loss of data, if the datacenter housing your instances is destroyed) option proposed there meets your needs. Without knowing your exact situation I would caution that it's easy to fall into the trap of overengineering these things and an entire modern datacenter going down is quite uncommon and the associated disaster (likely involving loss of power) may take other parts of our service offline anyway.

Here is the RabbitMq team blog I pulled the above quotes from. It is quite detailed and a good starting point for hardening your RabbitMq infrastructure.

Answer 2

Use Sqlite as your persisted outbound-queue.

You already have a conceptual outbound-queue from each device, which is implemented in memory (not persisted).

Sqlite is very very reliable. It has been pounded by billions of devices, and they have reached safety-critical certifications for the software. It also can practically run on almost every device. If you have limited disk space, you can quite easily run an SQL query to implement whatever data-trimming policy you design.

The internet itself is quite unreliable, particularly from the perspective of IoT devices. So the only way to avoid the biggest point of failure (prolonged internet unavailability) is to persist the outbound queue.

---

Even better if you can connect directly to the devices and query the sqlite database. Because the queue is already persisted, there's no need for an intermediate queue (RabbitMQ). (My company has plans to do this for our software, and to opensource the tooling).