Architecture for distributed packet inspection

Question

Currently I'm working on a new open-source project for a packet inspection. I have almost none experience with designing an architecture. Thus I would like to ask your opinion on my current architecture and what issues could I face during scaling it.

I have a set of large packet capture files with network traffic. These capture files are all captured on the same machine and follow each other in time. My primary task is to reconstruct all network flows that were happening in these files. Flows can span multiple files.

Here is what I have so far: I don't want to transfer files across a network as they are relatively large, therefore, I wrote a small web service that receives queries with the following parameters: file number, offset and amount of bytes. This send service responds with bytes from this location.

Another part is a separator. It starts from just a packet offset and does iteratively following steps:

1. Get bytes for next protocol header in this packet
2. Parse the header
3. Send information to update network flow information to a third component
4. Analyze what is the next protocol in protocol stack in the packet

Finally, the third part - flow re-constructor: It is a set of objects that contains some meta information about a flow(for example IP addresses, TCP ports) and a list of files with offset and size to describe flow’s payload.

My questions:

1. I feel like implementing “send service” and “re-constructor” by myself is a bad idea. What can be more optimal solution in terms of performance.
2. Currently I’m using rabbitMQ to communicate between components. Is there a more productive solution?
3. I was thinking to create multiple instances of separators(maybe on separate machines) so I need some mechanisms to orchestrate bytes delivery.
4. “Separator” waste much time just waiting for a delivery of bytes. How can I improve it?

Additionally, I highly appreciate any suggestions or comments on current architecture. Please help novice architect and open-source contributor:)

Answer 1

What you are proposing for your "send service" looks very much what a networked file system does.
This also uncovers a major flaw in the architecture: by separating the "send service" and "separator", you are required to transfer almost the entire capture files over the network. This is because the separator will be unable to skip large portions of the captured traffic.

To avoid having to transfer really large amounts of data over the network, you must do at the very least some initial packet inspection and filtering on the machine where the capture files reside. This initial filtering can already throw out all packets that are not interesting, because for example your friend's computer is not involved in the packet.

Additionally, as the capture files form a sequence in time, I would make use of that property and present the data in them to the logic that reconstructs frames and packets out of it as if there is just one really large capture file. The processing is way easier if only the file reader knows that there are multiple files and everything after it just sees one long stream of captured frames/bytes.

With that in mind, I would get to the following services

1. A packet service that can provide a stream/sequence of packets that match certain criteria. For example, address A (and possibly port P) is involved in the conversation; the conversation uses protocol X; etc.

   This service contains components to

   • read the capture files and present them as one long stream of capture data
   • extract packets from the capture data
   • filter packets based on the filter criteria

2. A conversation reconstruction service that can reconstruct a (HTTP) conversation out of multiple packets

3. A conversation analyzer that looks at the reconstructed conversations and looks for suspicious conversations (according to what you classify as suspicious).

As a final note, if the capture files are in a reasonably well-known format, then the chances are high that they can already be processed individually by tools like Wireshark. If reading the capture files over the network isn't that much of a problem, it might even be a possibility to patch such a tool that it can read from multiple consecutive files.