0

Let's say I have a system that parses network traffic, and generates event of things that happened. example:

  • ip1 was talking to ip2
  • ip2 was talking to ip3
  • etc

Now let's say that these events are published into event sourcing thing (like kafka for example).

I have another service (service B) that listens to these events, and keeps track of all IPs in the system. so that the work of service B is like:

ip = read_ip_from_event()
if not ip_exists(ip):
    add_new_ip(ip)

add_new_ip also publishes event (IpAddedEvent) in kafka.

Now here is my question: Let's say that I have a lot of traffic and I want to have multiple instances of service B. how can I make sure that eventually I get only one IpAddedEvent?

An issue can happen when ip doesn't exist, and two instances of serviceB reads from kafka (two different events with same ip), and both check ip_exists at the same time, and then both publish IpAddedEvent.

How can I avoid that, publishing only a single event?

7
  • You would need something to make the event unique, like IP address and event name. IP address would not be enough because other event could be sending you IP address data, so in this case its probably a combination of data elements. When the event is stored there is a unique constraint that avoid duplicates being added. Another consideration is tagging each event with a guid to make it unique and then other fields like event name, IP Address, Service Origin. Then technically there are different events, then just a query to dedup the IP addresses. That could work as well.
    – Jon Raynor
    Oct 5, 2021 at 15:50
  • 3
    Imo you’re trying to solve the wrong problem. Instead of trying to make the event unique, try to make the event-handler idempotent (able to handle the same event multiple times).
    – Rik D
    Oct 5, 2021 at 17:04
  • @JonRaynor do you mean that kafka (or similar solutions) knows not to store the same events twice, in case it has the same unique id? Oct 6, 2021 at 8:23
  • @RikD so I will have two IpAddedEvent, and whoever cares about ips would know to treat both events as single? I was thinking about the event stream as the source of truth, and that would be wrong. I mean, if the events describe things that happen in the network, then there was only single IpAddedEvent Oct 6, 2021 at 8:27
  • 1
    The event handler should be able to handle multiple notifications about a single event. You seem to think that the message is the event, but that’s not true; the event is the thing that happened, the message is informing you about the fact that it happened. It shouldn’t matter if you are informed multiple times that the same thing had happened.
    – Rik D
    Oct 6, 2021 at 12:28

2 Answers 2

1

The problem can be solved in different ways:

  1. When you call ip_exists(ip) you are essentially checking some data store for an IP. This data store should be consistent and distributed and should be accessible amongst multiple instances of service B to ensure that each call to ip_exists() will return the same result regardless of the instance.

  2. Your Kafka messages partitioning logic should ensure that only a subset of messages, perhaps messages within a given IP range go to one partition and within another IP range should land in another partition. This ensures that multiple instances of service B will partake of only a subset of IPs. This logic should be implemented at the producer side.

1
  • about answer 1: it won't help, because the checking and the creating of the event are not atomic. this means that multiple services can check that ip_exists (getting the same result), and then creating event, in the same time Dec 14, 2022 at 11:58
0

Since the Community Bot dug out this corpse, I'll try an alternative answer:

To ensure at most one IpAddedEvent it's actually easiest to have a single instance of service B. If you're bothered that this single instance would be a bottleneck, you can reduce its input feed frequency by pre-filtering.

The pre-filter processes would either be part of the original traffic sniffers, or they would listen to the message events created by the traffic sniffers, and keep a relatively small in-memory cache of recently seen IP addresses. Only for new IP addresses they generate an IpSeenEvent (for example through a private topic) that is processed by service B which then utilizes its long-term memory by virtue of ip_exists() to ensure that it only posts IpAddedEvent for actually new IP addresses. Since truly new IP addresses should be relatively rare, doing this in a single service worker should work fairly well.

2
  • Thank you. what you are basically saying is "You can't guarantee at most once event in that case", am I right? so we want to make serviceB single instance (or for example use some locks etc). Dec 14, 2022 at 11:57
  • 1
    Yes, that's the gist. You can do it with multiple instances of the service if you can use locking to ensure sequential processing, but then you don't gain much if anything relative to a single instance. Pre-filtering is the key to getting the amount of data down to be manageable by a single processing thread. Of course, you can also use sharding in addition if you have really massive amounts of data. Dec 14, 2022 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.