I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb.
My Question is how do I write the software part to be distributed. How do I best horizontally scale this. The obvious way would be to have multiple clients connected to IRC and each handle different channels. But I would i best split the work load?
I'm new to building software like this and was wondering what you guys thought would be the cleanest approach here. Do I just load like 100 channels per instance of my software and then maybe communicate via rabbitmq that I have joined these channels and nobody else needs to? How could I make it reduntant that maybe channels are always joined twice. (Messages have a unique Id from IRC, so there will be no issue in the DB of double entries, just insert errors maybe)
The main problems here is, how do I distribute "channels"/workloads between multiple clients the best.