I’m currently building a dashboard to view some analytics about the data generated by my company's product. We use MySQL as our database. The SQL queries to generate the analytics from the raw live data can be a bit complicated and take long time to process. So I put in place some batches that run every day or every hour, query this live data and generate the analytics and store this in some special tables that are queried only by the dashboard. It works well but the drawback is that the analytics are not real-time.
So I would like to know what is the best practice for my requirements. I don’t need strict real-time but near real-time of one or a few minutes.
I would like to know if replicating the live data from MySQL to something like hadoop or Elasticsearch would be a good solution.