I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time.

The APIs respond using aggregation queries. However when data to be aggregated is large, the response time of the API is too high.

What technology or methodology can I adopt to achieve low latency for API responses?

Current Approach

I am aggregating data using Spark Streaming based on the type of queries made by the API. This has reduced the API response time, but changes in queries take a long time to be reflected in API results as the whole data needs to be aggregated based on the new type of queries. But this leads to significant downtime. Is this the right approach. If yes, how can I lessen the downtime.

  • Caching works well if the requests are repeated often enough and the data doesn't have to be as fresh as the dew on a spring morning.
    – Neil
    Dec 22 '17 at 8:27
  • The data needs to be fresh and available even at the millisecond level.
    – nishant
    Dec 22 '17 at 8:28
  • Then there isn't much you can do. If you absolutely must have new data and you absolutely must aggregate large amounts of data, this is the only way. I would suggest you do some sort of background analysis on the data for quick access, but the data wouldn't be fresh. You could do caching, but again, the data wouldn't be fresh. You could do sampling of the data to reduce the workload, but then your results wouldn't be accurate. Speed / accuracy / memory -> pick two.
    – Neil
    Dec 22 '17 at 8:32
  • 2
    A speed and accuracy approach could be to keep an updated model in memory of aggregate data as your database is updated. Then you need not perform an analysis on the data, because your memory model is already accurate. However if you do it this way, you'd need to have a preload phase before you start serving requests that restores that memory model from the database. Also, you'd need to be very careful not to use up all the memory and have a plan for if and when you do.
    – Neil
    Dec 22 '17 at 9:03
  • 1
    MongoDB is already a no SQL database, is it not? Aren't we assuming that the current system is too slow? The preload phase is what happens when a program that heavily basis its aggregate info on what it has in memory must do before servicing requests. Otherwise memory is empty when program starts.
    – Neil
    Dec 22 '17 at 9:39

As I understand you you have a working system to serve responses to query “refreshes” but look for a way to be fast when a new “type” of query arrives. Unless you can predict what is to come then your only option is to add raw computing power. If you can predict some likely queries you may pre-populate the reply pipeline. Maybe you could have an Option for clients to warn what they will start requesting soon so preparation can start early giving a perceived better performance.

  • Sorry meant for this to be a comment not a reply but cannot delete it.
    – pco494
    Dec 22 '17 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.