1

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. My specific use case is I have about 4 different data streams coming into Kinesis, I combine them into an 'aggregate stream' after doing some processing, and then I store the result of that stream into a distributed DB for historical purposes. It's also emitted as another Kinesis stream for other consumers.

My goal is I want to have this final stream of data delivered to a web client in real time and have the client be able to see historical data of the stream. I have a couple questions regarding how I should design such a system.

  1. Should the API directly consume the Kinesis stream and relay the messages to the client? Or is it better to have a middle layer that sits between the Kinesis stream and the API client to manage some state.
  2. What is the recommended approach to manage replay? Suppose a client loses connection to the stream and returns 5 minutes later. How can I keep track of where the client left off so I can replay those events and then catch-up to the 'head' of the stream?
  3. Building on 2., should I have a sort of checkpoint mechanism on the stream where anything before the checkpoint would be in the distributed database and everything after that would be replayed?
  4. Should I even try to implement a replay functionality at all and rather just aggressively write to the distributed DB so that the distributed DB has a constant record of the true state? Then perhaps on the client a simple polling of the database could be done. I feel like this is kind of cheating though.

Thanks for your help!

Improve this question
1
  • You should make a separate post for each of your questions. I think that would result in better and more answers. – jhyot Mar 19 '18 at 8:57
1

Should the API directly consume the Kinesis stream and relay the messages to the client? Or is it better to have a middle layer that sits between the Kinesis stream and the API client to manage some state.

The answer to this hinges on the answers to the other questions.

What is the recommended approach to manage replay? Suppose a client loses connection to the stream and returns 5 minutes later. How can I keep track of where the client left off so I can replay those events and then catch-up to the 'head' of the stream?

Every stream consumer should keep track of what the offset / id / timestamp of the last successfully consumed message was. On restart, it will restart from the next offset / id / timestamp. When using Kafka this is either something you manage entirely yourself in the consumer code, or it is something the Kafka client SDK manages for you by storing consumed offsets in zookeeper (old client) or a kafka topic (new client). A kafka consumer client when reconnecting specifies the partition offsets from which it wants to start reading.

The benefit of this approach is that it covers the "read message but failed to process" problem scenario, and that it moves the complexity of managing consumer state into the consumers (which are free to choose to either restart from latest consumed offsets, or restart from latest, depending on the specific use case).

Improve this answer
3
  • Thank you for your response. I understand this part, but I'm also thinking one more layer above. The API is a consumer of the stream but the API itself serves many clients. So the API is kind of like fan-out if that makes sense. My question is more related to managing the state of the clients and replaying them individually. Suppose client A has an up to date representation but client B missed 5 minutes. Using only a single Kafka topic/kinesis stream, how can I ensure both clients will eventually 'catch up' to the head of the stream/topic? – coolboyjules Mar 19 '18 at 12:37
  • In that case I think your API has to also allow the clients to independently track which offset they are at and consume from the API at different offsets. Either by having each client announce itself with a client id and tracking offsets per client id, or by having each client keep track of its own offsets outside of the API. If you're looking towards an optimization where new messages are pushed to all clients all at once, then I don't think that fits with clients not being allowed to miss messages. – Joeri Sebrechts Mar 21 '18 at 8:12
  • You could just give each client a unique consumer group id and continue allowing Kafka to keep track of the offsets. – RubberDuck May 19 '18 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.