Architecture for distributed real time traffic capturing/viewer system

Question

I have a scenario as following (you can think of it as a distributed Wireshark system):

For a single capturing session, there are about 1 ~ 10 traffic capturing nodes distributed in a LAN network. Each capturing node will capture raw message data at speed about 1 ~ 1000 records/second for about 1 ~ 10 hour. In the meantime, there are several traffic viewer nodes (WPF app) in the same LAN network to view all message records captured.

Requirements:

1. All records of all capturing sessions must be persisted for further analysis.
2. During a capturing session, the viewer nodes should display the raw message and parsed details in semi real time (latency should be less than 1 minute since the raw message was captured).
3. On the viewer nodes, user can filter the data in each session easily (like Wireshark).

Now, I have a initial architecture as below:

And I can see several drawbacks in my initial architecture:

1. The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it.
2. Parsing message will take a lot of the CPU time of a viewer node, and each viewer node must parsing all messages separately which is a waste since the parsed result will be the same in all viewer nodes.
3. Hold all messages of a session can be a huge memory pressure (upper to 36M message, each message takes about 100~1000 byte) for the viewer node.

Any suggestions to improve the architecture design?

Answer 1

The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it.

Not to mention that you'll need an automated process to clean up the database.

I think that a better solution would use a streaming log such as Kafka. The idea of such a log is that you have multiple brokers, each writing to its own sequential message store. To increase bandwidth, you increase the number of brokers. You can configure replication (maybe not needed in your case) and also automatic sunsetting of records.

The big issue with such a stream is ordering: in general, records will be ordered only for one partition of the stream, and there is the potential for out-of-order writes even within a partition. But if you can apply an ordering at ingest, you can reconstruct the order on output.

the parsed result will be the same in all viewer nodes

This indicates to me that you would want a single machine that does analysis and produces a "cooked" output from the raw data. Since I would assume that you'd also want this output to be streaming, Kafka could be used for this case as well.

you can think of it as a distributed Wireshark system

Is this actually capturing network traffic? If yes then remember that you'll have to filter out the messages that you send to whatever is accumulating them.