Processing messages in parallel with Kafka & Akka

Question

We have data in Kafka which we need to export to multiple destinations. Each message key is to be exported to one destination.

A destination can be a REST endpoint, a file, a database etc.

Each exporter can have its own speed or rate limits and one exporter should not slow down the other.

In Kafka, the parallelism is dependent on the number of partitions, rather on the no. of messages.

Approach #1

We decided to use Akka where we read each message from the Kafka topic and tell to the exporter actors each of which will export to their respective destination like REST, file, database etc.

Problem: At-most once semantics only. The problem here is that, we have to commit the messages in Kafka. When we tell to an actor, we do not know, whether that actor has processed that message or not. It may still lie in the mailbox and we may commit that message. These committed messages are not read again after process restart.

while(true) {
    consumer.poll().forEach( record -> { 
        getExporter(record.key()).tell(record, ActorRef.noSender()));
    });
    consumer.commitAsync();
}

Approach #2

Read each message, store it in a persistent file, export it and remove it from the persistent file after export.

We need a persistent actor for this, for whom we need to tell to. So, we may use ask for this and wait till the actor puts it into the map and then tell it to the exporter.

Are there any better ways of doing this? Are there any reference architectures?

Answer 1

I don't know about reference architectures, but here are some more approaches:

Approach #3: at least once semantics

Hold the current batch of kafka messages in memory in the actor that reads from kafka, but do not commit immediately after polling. Mark the messages as handled when an actor sends a message back signaling export is complete, commit to kafka every time you have a batch of messages that is fully handled. Implement exporting in a way that is idempotent so that a restart causes some re-exporting but without this having bad consequences.

Approach #4: lock for export

Add an entry in a database for each message being processed. The exporter adds a record with its unique identifier signaling it is processing the message, and removes the record once done. Periodically scan the table for records which are stale, remove them and start a new exporter task for them. Exporters do not process messages which have entries in this table and do not belong to them.

Answer 2

If you want to process messages in parallel, Kafka already allows you to do that, you don't necessarily need Akka. The key to do this are consumer groups.

Basically a consumer group is a parallel set of consumers among which messages will be partitioned. Each group will however receive all the messages.

So what you could do, is define a group based on destination type (like REST, DB, etc.), and have multiple clients (even if in the same JVM) in parallel for each group.

This way:

• All destinations will receive all messages
• Each destination can scale on its own terms and rules without influencing the others
• Have at-least-once semantics by consuming the message only if the export was successful, again, without impacting performance

Each client is essentially a single thread that processes messages in a loop, so there's no need to use Akka. You just have to increase the number of clients as above.