Streaming Market Data Over The Public Internet

Question

I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this data is then processed and used to produce 3 distinct data sets as shown in the diagram below. I want to provide a web service that allows clients to connect through a gateway and ask for any of these feed types or all 3 through the same client.

Each Feed Handler is working with one distinct market data channel, so each channel corresponds to 3 outbound feeds and we have 30+ channels to consider. I'm looking to get some guidance about the best way to build such a system that can satisfy the following requirements.

• Clients can use a single SDK to connect and consume market data across various channels.
• I want to control what kinds of data clients can consume for billing purposes. To make things more challenging, clients might only have access to some products within a single dataset e.g. streaming Top Of Book for product ABC on channel 1.

The biggest design decisions I have right now, is how to store and service clients from one or many publishers. My initial plan was to build an SDK that connects to one or many of these data publisher nodes and have the publisher push data down a TCP socket to the client. This works fine for one channel but it becomes a problem when customers want to consume data from multiple channels. Let's say a client wants to consume data from channel 1 and 2, that means opening 2 socket connections and multiplexing across both sockets. As you can imagine, we might have clients who want to consume from many channels at the same time.

I've also considered pushing data from each channel into a single fleet of publishers ignoring the channel abstraction completely but this data is very high volume and message latencies would be a concern. Data on one channel might swamp a single channel and make the solution unscalable.

Is TCP the correct approach here? I've also considered exposing this data through GCP pub/sub and letting clients consume from distinct feeds but this is abit of a maintence nightmare because we would end up with many topics partitioning the different feed types per channel. This would also couple us to a single pub/sub stack in GCP.

I hope this doesn't sounds like an open ended question, if so let me know. Any advice is appreciated.

Answer 1

Your architecture should be driven by requirements. The basic idea that you will e.g. shovel updates down a socket, is not the architecture.

• Latency - how fast do you have to get updates to the client?

  If you have a multi-layered architecture each layer will add latency. Pub/sub has latency. Queues waiting for GCF or other web services add latency.

• Throughput - how many updates per second do you need to support?

  How many CPU microseconds does it take to process one update?

• Simultaneous users - How many users do you need to support?

  If you need to scale out to tens of thousands of users you need a way to do this.

• Simultaneous subscriptions per user - How many do you need to support?

• Cost - How much can it cost per user per hour?

  You need to relate this to all of the above. What is the cost per CPU/minute? Per RAM GB/minute? Per I/O GB?

You need answers to these questions from the client. If necessary, you can make up reasonable numbers and propose them to the client, but you need them to agree to some set of numbers or you run the real risk that you build something that just cannot do the job the client wants it to do.