Currently I'm working on a new open-source project for a packet inspection. I have almost none experience with designing an architecture. Thus I would like to ask your opinion on my current architecture and what issues could I face during scaling it.
I have a set of large packet capture files with network traffic. These capture files are all captured on the same machine and follow each other in time. My primary task is to reconstruct all network flows that were happening in these files. Flows can span multiple files.
Here is what I have so far: I don't want to transfer files across a network as they are relatively large, therefore, I wrote a small web service that receives queries with the following parameters: file number, offset and amount of bytes. This send service responds with bytes from this location.
Another part is a separator. It starts from just a packet offset and does iteratively following steps:
- Get bytes for next protocol header in this packet
- Parse the header
- Send information to update network flow information to a third component
- Analyze what is the next protocol in protocol stack in the packet
Finally, the third part - flow re-constructor: It is a set of objects that contains some meta information about a flow(for example IP addresses, TCP ports) and a list of files with offset and size to describe flow’s payload.
- I feel like implementing “send service” and “re-constructor” by myself is a bad idea. What can be more optimal solution in terms of performance.
- Currently I’m using rabbitMQ to communicate between components. Is there a more productive solution?
- I was thinking to create multiple instances of separators(maybe on separate machines) so I need some mechanisms to orchestrate bytes delivery.
- “Separator” waste much time just waiting for a delivery of bytes. How can I improve it?
Additionally, I highly appreciate any suggestions or comments on current architecture. Please help novice architect and open-source contributor:)