I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb.

My Question is how do I write the software part to be distributed. How do I best horizontally scale this. The obvious way would be to have multiple clients connected to IRC and each handle different channels. But I would i best split the work load?

I'm new to building software like this and was wondering what you guys thought would be the cleanest approach here. Do I just load like 100 channels per instance of my software and then maybe communicate via rabbitmq that I have joined these channels and nobody else needs to? How could I make it reduntant that maybe channels are always joined twice. (Messages have a unique Id from IRC, so there will be no issue in the DB of double entries, just insert errors maybe)

The main problems here is, how do I distribute "channels"/workloads between multiple clients the best.

  • 1
    chose one problem and ask about that! – Ewan Aug 7 '18 at 7:58
  • The main problems here is, how do I distribute "channels"/workloads between multiple clients the best. – gempir Aug 7 '18 at 8:04
  • if the nature of the workload is writing to the db. you cant – Ewan Aug 7 '18 at 8:07
  • Why not? Can you explain your point? I see no issue with multiple clients writing different messages to 1 DB. – gempir Aug 7 '18 at 8:08
  • exactly to one DB, its the DB doing the work, not your client – Ewan Aug 7 '18 at 8:08

To distribute a channel list amongst various clients, simply have a database table

table Channel
    varchar(50) id
    varchar(50) name
    varchar(50) clientId

Populate the table with the channels you wish to monitor and fill in the clientId with the client you want to monitor that channel.

If possible, have the program that populates the table also fire up instances of the client as required, injecting, or reading its Id as it starts up

Each client, will check the table for channels that it should be monitoring and initiate a connection.

A monitor program will monitor the clients and reassign their channels if they crash.

  • Alright. This sounds like a solid plan. The monitor program is key here IMO. Good Solution. Thank you so much – gempir Aug 7 '18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.