3

I have an application (run by a Windows Service) which connects to a TCP server (let's call the service 'Listener' from here on).

[It's important to note that the TCP server is out of my reach, and should be considered as an external component that cannot be changed in regard to the following question]

The TCP Server pushes data to any client that is connected to it, and this data starts a chain of complex business logic, which is very demanding.

In order to improve reliability, I would like to run several Listeners, and distribute the incoming data between them.

They share a distributed cache (Redis, if it matters).

How would you design an algorithm that is effective, fast (as time is of the essence) and the different Listeners should not be dependent on one another (should one of them fail, the algorithm should still work)?

It is also important that no server data goes down the drain (e.g. It shouldn't be possible for no Listener to deal with server-sent data)

I was thinking in the direction of Leader Election based on random numbers - am I in the right direction?

[Listener is written in C#]

2

TCP is not a great protocol for this kind of thing.

I would suggest a middle layer. A single TCP listener that receives all messages and puts them on a queue, where "a queue" means any queuing system that allows distribution of messages across multiple subscribers.

RabbitMQ is pretty easy to implement in .NET and very flexible for your needs. Work Queues are the concept you're looking for.

  • What happens when that middle layer fails for any reason? I was thinking of all Listeners 'adding' to the queue if the message still isn't in the queue or something of the sort - would that work? Also, can I implement some kind of Work Queue using Redis, without needing RabbitMQ? – Berlo Feb 19 '16 at 10:41
  • @Berio: There are literally no ways to guarantee a listener is always listening from the side of the listener. It's the responsibility of the sender to check that the message got received. If you have no control over that, you have a problem, and the best possible solution is to make the listener super-simple, so it almost never fails (and have auto-recovery, which Windows offers for services). And yes, Redis would also work, I believe ... "any queuing system that allows distribution of messages across multiple subscribers". – pdr Feb 19 '16 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.