0

I have a scenario as following (you can think of it as a distributed Wireshark system):

For a single capturing session, there are about 1 ~ 10 traffic capturing nodes distributed in a LAN network. Each capturing node will capture raw message data at speed about 1 ~ 1000 records/second for about 1 ~ 10 hour. In the meantime, there are several traffic viewer nodes (WPF app) in the same LAN network to view all message records captured.

Requirements:

  1. All records of all capturing sessions must be persisted for further analysis.
  2. During a capturing session, the viewer nodes should display the raw message and parsed details in semi real time (latency should be less than 1 minute since the raw message was captured).
  3. On the viewer nodes, user can filter the data in each session easily (like Wireshark).

Now, I have a initial architecture as below:

enter image description here And I can see several drawbacks in my initial architecture:

  1. The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it.
  2. Parsing message will take a lot of the CPU time of a viewer node, and each viewer node must parsing all messages separately which is a waste since the parsed result will be the same in all viewer nodes.
  3. Hold all messages of a session can be a huge memory pressure (upper to 36M message, each message takes about 100~1000 byte) for the viewer node.

Any suggestions to improve the architecture design?

  • Like the other answer said, a database is not the appropriate tool to communicate this information. I would consider using WCF or some tcp/ip based library for fast streaming, and I would write the data to files directly, where you have full control over optimization opportunities. – Frank Hileman Mar 16 '18 at 4:01
  • Good point. Maybe I'll store basic information into database and store the data into files. – ricky Mar 16 '18 at 4:51
2

The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it.

Not to mention that you'll need an automated process to clean up the database.

I think that a better solution would use a streaming log such as Kafka. The idea of such a log is that you have multiple brokers, each writing to its own sequential message store. To increase bandwidth, you increase the number of brokers. You can configure replication (maybe not needed in your case) and also automatic sunsetting of records.

The big issue with such a stream is ordering: in general, records will be ordered only for one partition of the stream, and there is the potential for out-of-order writes even within a partition. But if you can apply an ordering at ingest, you can reconstruct the order on output.

the parsed result will be the same in all viewer nodes

This indicates to me that you would want a single machine that does analysis and produces a "cooked" output from the raw data. Since I would assume that you'd also want this output to be streaming, Kafka could be used for this case as well.

you can think of it as a distributed Wireshark system

Is this actually capturing network traffic? If yes then remember that you'll have to filter out the messages that you send to whatever is accumulating them.

  • 1. Yes, clean up the database will be a problem since it grows rapidly. – ricky Mar 15 '18 at 12:14
  • 2. I'm thinking of use a dedicated node to parsing the raw data and store it somewhere for reading. 3. It's for industry specific device network, not for the general TCP/IP network. 4. I think use a solution like Kafka will remarkably increase the complexity of the whole software system. And still trying to find if there is a simper solution. Thanks very much! – ricky Mar 15 '18 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.