I have a C#-based system that works relatively well. It reads data from kafka, processes the data and then writes it out to MSSQL.
The kafka topic has been partitioned into 10 partitions - and after 2 years, the team is considering increasing the number of partitions because the cluster is showing signs of not managing the volumes anymore. (Volumes are large and increase every month).
My consumers have been implemented as windows services, where each service starts up one kafka consumer. We've installed 10 services - so that each service handles one partition.
This design works well and if 5 of the services are down, the consumer rebalances and the 5 remaining services will each handle 2 partitions.
When the topics increase, I should be able to install and start a few extra consumers and everything should continue smoothly.
I'm starting to question whether I shouldn't change the design to have a single service start multiple KafkaReaders (basically multiple threads, each with a kafka consumer). If I do this - scaling would be easier, just configure the service to start more threads. (I would probably still run at least 2 services for redundancy, or in case I want to take one down.)
I have heard anecdotes though where people claim that multiple threads in the same service is slower on the kafka consume than multiple applications. I don't see why this would be the case, and in my case, I can get some nice savings in memory if I consolidate into a single service.
Is there any logical reason that you know of why running 10 threads in a single service would perform worse than running 10 services?
Either I'm missing some basic principle - or the anecdotes were just... inaccurate(?)...