We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation.
We are looking for an efficient, scalable way to store this huge volume, such that aggregations can be queried by business logic shortly after storage, and such that they can be aggregated into dashboards as well.
We realize that basing business logic on metrics is not typical. Usually, when faced with data that needs to be both written and read by an application, we would choose a typical database such as Postgres or MongoDB. However, the volume we want to deal with has made us think twice about our choice. Here are some options we've considered, which may or may not be a good fit:
- Time-series databases such as Graphite or InfluxDB, which is great for write-heavy workloads even at this volume, but might not be so good when retrieving data;
- Redis, which can handle very fast reads and writes, but may struggle with the amount of swaps between storage and memory which would occur with this volume;
- Mongo DB, which is a little expensive and would create one document per measurement, making the database grow into the billions of documents range in a few days;
- Redshift (we are on AWS).
Is any of these options objectively better than the others (and why/why not)? Or are there any other options out there which would allow for this sort of efficient reads and writes over billions of datapoints?
Remark: There is a similar question here. But it's from 2015 and seems outdated and not comprehensive enough.