I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. My specific use case is I have about 4 different data streams coming into Kinesis, I combine them into an 'aggregate stream' after doing some processing, and then I store the result of that stream into a distributed DB for historical purposes. It's also emitted as another Kinesis stream for other consumers.
My goal is I want to have this final stream of data delivered to a web client in real time and have the client be able to see historical data of the stream. I have a couple questions regarding how I should design such a system.
- Should the API directly consume the Kinesis stream and relay the messages to the client? Or is it better to have a middle layer that sits between the Kinesis stream and the API client to manage some state.
- What is the recommended approach to manage replay? Suppose a client loses connection to the stream and returns 5 minutes later. How can I keep track of where the client left off so I can replay those events and then catch-up to the 'head' of the stream?
- Building on 2., should I have a sort of checkpoint mechanism on the stream where anything before the checkpoint would be in the distributed database and everything after that would be replayed?
- Should I even try to implement a replay functionality at all and rather just aggressively write to the distributed DB so that the distributed DB has a constant record of the true state? Then perhaps on the client a simple polling of the database could be done. I feel like this is kind of cheating though.
Thanks for your help!