I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this data is then processed and used to produce 3 distinct data sets as shown in the diagram below. I want to provide a web service that allows clients to connect through a gateway and ask for any of these feed types or all 3 through the same client.
Each Feed Handler is working with one distinct market data channel, so each channel corresponds to 3 outbound feeds and we have 30+ channels to consider. I'm looking to get some guidance about the best way to build such a system that can satisfy the following requirements.
- Clients can use a single SDK to connect and consume market data across various channels.
- I want to control what kinds of data clients can consume for billing purposes. To make things more challenging, clients might only have access to some products within a single dataset e.g. streaming Top Of Book for product ABC on channel 1.
The biggest design decisions I have right now, is how to store and service clients from one or many publishers. My initial plan was to build an SDK that connects to one or many of these data publisher nodes and have the publisher push data down a TCP socket to the client. This works fine for one channel but it becomes a problem when customers want to consume data from multiple channels. Let's say a client wants to consume data from channel 1 and 2, that means opening 2 socket connections and multiplexing across both sockets. As you can imagine, we might have clients who want to consume from many channels at the same time.
I've also considered pushing data from each channel into a single fleet of publishers ignoring the channel abstraction completely but this data is very high volume and message latencies would be a concern. Data on one channel might swamp a single channel and make the solution unscalable.
Is TCP the correct approach here? I've also considered exposing this data through GCP pub/sub and letting clients consume from distinct feeds but this is abit of a maintence nightmare because we would end up with many topics partitioning the different feed types per channel. This would also couple us to a single pub/sub stack in GCP.
I hope this doesn't sounds like an open ended question, if so let me know. Any advice is appreciated.