Architecture to populate graphs from real-time data

Question

I'm developing an application where graphs need to be populated from realtime data.
The Real-Time data comes from a Kafka Queue.

How should I send this real-time data to the front-end which is in Angular8?
So far I have thought of two approaches:

1. Use a push-based queue mechanism from the server over the Kafka queue, and then transfer the data to the Front-end via Server-Sent Events (SSE) whenever new data is fetched from the queue (since this will not be duplex communication).

2. Use a push-based Kafka queue and subscribe to it from Angular itself.

I know web-scockets can be an option, but I don't need a duplex communication. Data will only be sent from the server.

Is this the right approach? Or is there any other better way
Also, is it a good practise to subscribe to a queue from the front-end?

I'm using Django for the back-end and Angular for the front-end

Answer 1

I'm not sure how Kafka Queue works. But you might want to consider for the back-end using Django REST Framework. Then you can access the data using JSON to Angular to display the results.

If so you will need to create a model in a Django app to describe what kind of data that should be stored in the database. When that is done, you need to configure up the Django REST framework what model should be accessed through the API. The documentation on their website is very good.

Second you need to setup a connection between Apache Kafka and Python/Django. You could check out these projects for that:

• Kafka Python client
• PyKafka

Then you will need to create a Django Command to populate the database with the model that you created. And you could add a cronjob to run that command how often as you would like to fetch the new data.

Answer 2

Lacking a need for duplex communication is not a reason to avoid web sockets, or any other solution. Your choice of technology should balance these needs:

• Quick development
• Ease of maintenance
• Availability of code libraries on the server and client side to handle this communication for you
• How easy it would be to switch out your web front end framework and the queue technology.

I would be leery of exposing your Kafka queue to the client, which means that queue would be available publically. I would at least have a small server application running that connects to the queue so it can enforce authentication and authorization, as well as reformatting the data from the queue making it easier to work with on the client in JavaScript. This decouples the client (an AngularJS application) from the data source (your Kafka queue), which gives you flexibility for the future should you ever need to change out the source for these events.