I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time.
The APIs respond using aggregation queries. However when data to be aggregated is large, the response time of the API is too high.
What technology or methodology can I adopt to achieve low latency for API responses?
I am aggregating data using Spark Streaming based on the type of queries made by the API. This has reduced the API response time, but changes in queries take a long time to be reflected in API results as the whole data needs to be aggregated based on the new type of queries. But this leads to significant downtime. Is this the right approach. If yes, how can I lessen the downtime.