0

Let's say I have a system that parses network traffic, and generates event of things that happened. example:

  • ip1 was talking to ip2
  • ip2 was talking to ip3
  • etc

Now let's say that these events are published into event sourcing thing (like kafka for example).

I have another service (service B) that listens to these events, and keeps track of all IPs in the system. so that the work of service B is like:

ip = read_ip_from_event()
if not ip_exists(ip):
    add_new_ip(ip)

add_new_ip also publishes event (IpAddedEvent) in kafka.

Now here is my question: Let's say that I have a lot of traffic and I want to have multiple instances of service B. how can I make sure that eventually I get only one IpAddedEvent?

An issue can happen when ip doesn't exist, and two instances of serviceB reads from kafka (two different events with same ip), and both check ip_exists at the same time, and then both publish IpAddedEvent.

How can I avoid that, publishing only a single event?

7
  • You would need something to make the event unique, like IP address and event name. IP address would not be enough because other event could be sending you IP address data, so in this case its probably a combination of data elements. When the event is stored there is a unique constraint that avoid duplicates being added. Another consideration is tagging each event with a guid to make it unique and then other fields like event name, IP Address, Service Origin. Then technically there are different events, then just a query to dedup the IP addresses. That could work as well.
    – Jon Raynor
    Oct 5, 2021 at 15:50
  • 2
    Imo you’re trying to solve the wrong problem. Instead of trying to make the event unique, try to make the event-handler idempotent (able to handle the same event multiple times).
    – Rik D
    Oct 5, 2021 at 17:04
  • @JonRaynor do you mean that kafka (or similar solutions) knows not to store the same events twice, in case it has the same unique id? Oct 6, 2021 at 8:23
  • @RikD so I will have two IpAddedEvent, and whoever cares about ips would know to treat both events as single? I was thinking about the event stream as the source of truth, and that would be wrong. I mean, if the events describe things that happen in the network, then there was only single IpAddedEvent Oct 6, 2021 at 8:27
  • 1
    The event handler should be able to handle multiple notifications about a single event. You seem to think that the message is the event, but that’s not true; the event is the thing that happened, the message is informing you about the fact that it happened. It shouldn’t matter if you are informed multiple times that the same thing had happened.
    – Rik D
    Oct 6, 2021 at 12:28

1 Answer 1

0

The problem can be solved in different ways:

  1. When you call ip_exists(ip) you are essentially checking some data store for an IP. This data store should be consistent and distributed and should be accessible amongst multiple instances of service B to ensure that each call to ip_exists() will return the same result regardless of the instance.

  2. Your Kafka messages partitioning logic should ensure that only a subset of messages, perhaps messages within a given IP range go to one partition and within another IP range should land in another partition. This ensures that multiple instances of service B will partake of only a subset of IPs. This logic should be implemented at the producer side.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.