I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this data is then processed and used to produce 3 distinct data sets as shown in the diagram below. I want to provide a web service that allows clients to connect through a gateway and ask for any of these feed types or all 3 through the same client.

Each Feed Handler is working with one distinct market data channel, so each channel corresponds to 3 outbound feeds and we have 30+ channels to consider. I'm looking to get some guidance about the best way to build such a system that can satisfy the following requirements.

  • Clients can use a single SDK to connect and consume market data across various channels.
  • I want to control what kinds of data clients can consume for billing purposes. To make things more challenging, clients might only have access to some products within a single dataset e.g. streaming Top Of Book for product ABC on channel 1.

The biggest design decisions I have right now, is how to store and service clients from one or many publishers. My initial plan was to build an SDK that connects to one or many of these data publisher nodes and have the publisher push data down a TCP socket to the client. This works fine for one channel but it becomes a problem when customers want to consume data from multiple channels. Let's say a client wants to consume data from channel 1 and 2, that means opening 2 socket connections and multiplexing across both sockets. As you can imagine, we might have clients who want to consume from many channels at the same time.

I've also considered pushing data from each channel into a single fleet of publishers ignoring the channel abstraction completely but this data is very high volume and message latencies would be a concern. Data on one channel might swamp a single channel and make the solution unscalable.

Is TCP the correct approach here? I've also considered exposing this data through GCP pub/sub and letting clients consume from distinct feeds but this is abit of a maintence nightmare because we would end up with many topics partitioning the different feed types per channel. This would also couple us to a single pub/sub stack in GCP.

I hope this doesn't sounds like an open ended question, if so let me know. Any advice is appreciated.

High Level Data Flow

  • Your primary concerns should be latency and availability. I.e. how many milliseconds can a client wait for an update? After how many milliseconds should the client conclude the network is down and data is no longer good?
    – Ben
    Dec 28, 2021 at 1:03
  • In an internal network, latencies should be under 100ms, this service is not intended for real time trading because there is no multicast in the public cloud, so we are relying on moving the data into GCP and using GCP pub sub to replicate it across various regions. I think predictable latencies of 100ms in a public cloud might be difficult and anything around 500ms would be more likely. I really don't know.
    – john
    Dec 28, 2021 at 9:18
  • My point is you have to decide what latency etc. you want and design to that. It has to be architected in. If the client hasn't given you a requirement, you must suggest one and put it to the client, and make sure they are happy with it.
    – Ben
    Dec 28, 2021 at 11:06
  • Otherwise you may discover e.g. that you cannot get the latency you need using pub/sub and the whole thing has to be thrown away and re-done.
    – Ben
    Dec 28, 2021 at 11:06
  • "Let's say a client wants to consume data from channel 1 and 2, that means opening 2 socket connections and multiplexing across both sockets." You lost me here. What are you multiplexing exactly? The three feeds? Are you overcomplicating this, maybe?
    – JimmyJames
    Dec 28, 2021 at 21:58

1 Answer 1


Your architecture should be driven by requirements. The basic idea that you will e.g. shovel updates down a socket, is not the architecture.

  • Latency - how fast do you have to get updates to the client?

    If you have a multi-layered architecture each layer will add latency. Pub/sub has latency. Queues waiting for GCF or other web services add latency.

  • Throughput - how many updates per second do you need to support?

    How many CPU microseconds does it take to process one update?

  • Simultaneous users - How many users do you need to support?

    If you need to scale out to tens of thousands of users you need a way to do this.

  • Simultaneous subscriptions per user - How many do you need to support?

  • Cost - How much can it cost per user per hour?

    You need to relate this to all of the above. What is the cost per CPU/minute? Per RAM GB/minute? Per I/O GB?

You need answers to these questions from the client. If necessary, you can make up reasonable numbers and propose them to the client, but you need them to agree to some set of numbers or you run the real risk that you build something that just cannot do the job the client wants it to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.