2

Background

We have a ton of GPS devices in vehicles. These devices need to communicate with our system. To achieve this, we need to parse the device's messages before proceeding.

The following describes my scalable and fail resistant architecture idea.

Architecture

The devices are black boxes that send us buffers of data via TCP every X seconds. A typical message follows this trajectory:

  1. Each device communicates with a RabbitMQ server (R1) that has a persistent queue. This way if something fails, no messages are lost.

  2. R1 sends the messages to the Message Parser Cluster (Hamster Cluster). Fancy name for a bunch of hamsters that receive a data buffer as input and output a JSON object we can understand.

  3. The Hamster Cluster then sends the parsed messages to another Persistent Queue (R2), which has the same properties as R1.

  4. R2 then sends these messages to the Data Processing Cluster (Monkey Cluster) which does the real work. We are inside our system now. The path ends.

architecture_design

Objectives

The main objectives here are to have an architecture design with the following properties:

  1. Scalable. We must be able to recruit more Hamsters and Monkeys if any of our Clusters is dying ( No one likes dead animals! )

  2. Fail proof. If any given machine in the system fails, the service must not go down.

Problems/Questions

  1. As designed, this system has 2 critical failure points: R1 and R2. Should this machines fail, the whole system goes down. Is there a way to avoid having these two critical failure points?

  2. My coworkers made the case for NGINX, which also supports TCP/UDP connections. I understand that using it, we would have load balancing between the machines of each cluster. However, Replacing the Rabbit servers for NGINX servers would still have 2 points of critical failure. Could this be avoided?

  • Are you searching for general architecture solution or system/networks one ? On the network side, the simple answer would be to have a each rabbit server double by a fail-over, given that the R2 and monkey won't bother the fact that you might have messages not ordered when they come into R2 or the data processing cluster. If ordering is required, you will necessary need a single point gathering the data and so a failure point. Also if you architecture is behind a router, that router will also be a single failure point. Others solution would involve cache on the device and hamster side. – Walfrat Apr 23 '18 at 11:32
  • Given your description I would go the system/network solution route.The single entry point is R1 and we don't really care about order at any given step, provided that the time difference is not abysmal ( not greater than X minutes, for example. Each message is parsed rather quickly, so I don't see this happening ). Would you care to elaborate on your cache suggestion? I fail to see how it would help. – Flame_Phoenix Apr 23 '18 at 11:44
  • Well the cache solution would simply be able to store locally data and push them later in the case you still have a single point of failure, that is suppose your requirments make it so you shouldn't loss any datas or be able to at least have a history tracking one information every X minutes. – Walfrat Apr 23 '18 at 11:56
  • 2
    RabbitMQ is normally deployed in clusters of 3 or more nodes for high availability. The single point of failure I can see is the ingress point / network connection between the devices and R1. I guess you could either expose multiple nodes in the cluster, or maybe use a load balancer – Justin Apr 23 '18 at 12:16
  • Still, here we have the same issue. If I use the Load Balancer, then the Load Balancer becomes critical point of failure. If I create a Rabbit Cluster then the master Node ( exposed to the outside world ) becomes the point of failure. – Flame_Phoenix Apr 23 '18 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.