How to store a huge volume of time-series datapoints in an efficient way?

Question

We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation.

We are looking for an efficient, scalable way to store this huge volume, such that aggregations can be queried by business logic shortly after storage, and such that they can be aggregated into dashboards as well.

We realize that basing business logic on metrics is not typical. Usually, when faced with data that needs to be both written and read by an application, we would choose a typical database such as Postgres or MongoDB. However, the volume we want to deal with has made us think twice about our choice. Here are some options we've considered, which may or may not be a good fit:

• Time-series databases such as Graphite or InfluxDB, which is great for write-heavy workloads even at this volume, but might not be so good when retrieving data;
• Redis, which can handle very fast reads and writes, but may struggle with the amount of swaps between storage and memory which would occur with this volume;
• Mongo DB, which is a little expensive and would create one document per measurement, making the database grow into the billions of documents range in a few days;
• Redshift (we are on AWS).

Is any of these options objectively better than the others (and why/why not)? Or are there any other options out there which would allow for this sort of efficient reads and writes over billions of datapoints?

Remark: There is a similar question here. But it's from 2015 and seems outdated and not comprehensive enough.

Answer 1

With this kind of load I suggest you roll your own solution using a compiled language and you use the operating system's file system directly as a storage medium.

Your requirements seem pretty simple (although you are not very specific). You could structure your data in folders that map to the calendar and the clock, putting 2022 data in one folder, February's data in it's own subfolder, data of the 17th in another subfolder. Then start a new file every minute.

By doing so you may leave out the timestamp if points are coming in at a fixed rate (the time may be derived from the point data's index). This may not be the case though, it is just one idea to optimize the use of space.

Depending on the meta data you need to store with your points you can design a record layout that makes best use of a series of bits.

Searching and reading would be trivial, you could use the same classes and structures used for storing. That is, if searching based on time would do. If not and you want to be able to search based on some of that other undisclosed meta data, you would need to produce indexes which would complicate matters considerably.

Do you need to hot-swap disks or could you fit several sessions on a single disk? There are still a lot of questions to be answered.

How bad are 10K data points per second really? Say you need 100 bytes per point (seems generous), that would be 1 MB/s, just under 100 GB a day. One 10 TB drive should give you 100 days of storage.

Compression could extend this to several years but would complicate searching and reading a bit.