0

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For each update, an event is fired which is sent to Kafka.

There can be multiple consumers deployed on different machines and they can scale up or down as per load. The consumers consume and process the events and save changes in a scalable and efficient database.
Order of events is important for me. For example, I get a product-create event. A product P is created and lies in category C. It is important that event for the creation of category C is processed before the product-create event for product P. Now if there are two consumers, and one consumer picks up product-create event for product P and the other consumer picks up event for creation of category C, it may happen product-create event is processed first, which will lead to data inconsistency.
There can be multiple such dependencies. How do I ensure the ordered processing or some alternative to ensure data consistency?

Two solutions that are right now in my mind:

  1. We can re-queue an event until its dependent event is successfully processed.
  2. We can wait for the dependent event to get processed and try processing the event at some intervals say 1 second with some maximum retries.

Requeuing has issues that event is now stale and no longer required. Example:

  • Initial Order = Create-Event(Dependent on event X), Event X, Delete-Event .
  • After Requeuing, Order = Event X, Delete-Event, Create-Event(Dependent on event X).
    Create event is processed after delete event again leading to inconsistent data.

The same issue is applicable to the second solution (waiting and retrying).

Above issues can be solved by maintaining versions for events and ignoring an event if the targeted object(which is going to be modified by the event) has a higher version than that of the event.
But I am very unsure of the pitfalls and the challenges of the above solutions that might not be very obvious right now.

PS: Stale data works for me but there should be no inconsistencies.

2

I believe Kafka has the notion of "Partitions". If you put messages in the same partition they will be processed in order by a single processor

"By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order"

Also, this post has some interesting detail on the topic

https://stackoverflow.com/questions/38024514/understanding-kafka-topics-and-partitions

HOWEVER!!

This won't help you at all if the events themselves are created in the wrong order. Which, unless you are checking whether a category exists before someone clicks the create product button is going to be problematic.

If your create category event is slow enough to need to be async, then a better solution I think would be to batch the events in the UI before sending them to be processed.

ie. I click create category, nothing is sent, I just start building a batch of messages, I click create product in that category, another message is added to the batch.

When I click "Save" the whole batch is sent as a single message to be processed as one thing, rather than many individual things, with appropriate transactions and locking to preserve data consistency.

  • I don't have a single processor. This is the main problem. The ordering of events is ensured in Kafka. But as there are multiple instances of consumers, the order of processing is now no more guaranteed. Kafka docs also say that if the number of consumer instances is less than partitions, a consumer will receive events from multiple partitions. Consumer being an application can die anytime. – rohanagarwal Jan 2 at 14:45
  • the important thing is that the same consumer gets all the messages in the correct order and hence they will be processed in the order that they are enqueued – Ewan Jan 2 at 14:50
  • so, it is going to be like this - I create multiple partitions for a topic, assign one consumer to each partition, which makes the solution scalable. If a consumer goes down, its partition will be assigned to some other consumer. Events for one product will always go to one partition(events will not be scattered across partitions for a particular product). – rohanagarwal Jan 3 at 10:44
  • what I did is - I created different topics for category, channels, brands, product and bundled products. Product and product bundling events are huge, so there are multiple partitoons for that topic and one consumer for each partiton. I always process category(categroy del event may still cause inconsistencies) topic first upto a certain timestamp, then go for channels and brands, then products, then bundled products upto the same timestamp. Different products don't have any dependencies between them except bundling for which I have a different topic and I procss it at last. – rohanagarwal Jan 3 at 10:52
2

The very point of asynchronous systems such as Kafka is that you gain performance in exchange for some kind of guarantees - in your case the guarantee for in-order processing. Forcing one event to wait for another pretty much removes the benefits Kafka offers to you - if you want to go that way, better use a queueing system instead of an asynchronous one right from the start.

So, the solution which lets you keep the advantages of asynchronous processing (at the cost of code complexity) is to introduce versioning for each of your events. This will make the right ordering of events pertaining to a single entity possible: receive event, check if version is newer than the one already in database, if newer - overwrite, if not newer - ignore. Note that you can also receive exactly the same version more than once due to errors and retries. For records pertaining to different entities, the solution will depend on how you process them. You could save the received events to a local DB independently for each entity (with versioning), and then once in a while run a process which would combine the data. Then, for example if there is already a record for a product but not yet for the matching category, just ignore the product. During the next run, the category's record will probably already be there and you will be able to process the product.

There are also solutions such as Kafka Streams.

One way or another, consider if you really need an asynchronous system such as Kafka since their performance benefits can only be reaped at the cost of added complexity and in some situations you may be better off using a simpler, synchronous system.

  • the only problem I have using a single computer is high availability, I don't see scalability as much of a concern in my case, also can you shed more light on Kafka streams? – rohanagarwal Jan 3 at 5:48
  • 1
    @rohanagarwal Kafka Streams are described here: kafka.apache.org/21/documentation/streams/core-concepts I haven't used it in practice, but the descriptions sounds like it could be helpful in your usage scenario. – Michał Kosmulski Jan 3 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.