I have a scenario as following (you can think of it as a distributed Wireshark system):
For a single capturing session, there are about 1 ~ 10 traffic capturing nodes distributed in a LAN network. Each capturing node will capture raw message data at speed about 1 ~ 1000 records/second for about 1 ~ 10 hour. In the meantime, there are several traffic viewer nodes (WPF app) in the same LAN network to view all message records captured.
Requirements:
- All records of all capturing sessions must be persisted for further analysis.
- During a capturing session, the viewer nodes should display the raw message and parsed details in semi real time (latency should be less than 1 minute since the raw message was captured).
- On the viewer nodes, user can filter the data in each session easily (like Wireshark).
Now, I have a initial architecture as below:
And I can see several drawbacks in my initial architecture:
- The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it.
- Parsing message will take a lot of the CPU time of a viewer node, and each viewer node must parsing all messages separately which is a waste since the parsed result will be the same in all viewer nodes.
- Hold all messages of a session can be a huge memory pressure (upper to 36M message, each message takes about 100~1000 byte) for the viewer node.
Any suggestions to improve the architecture design?