0

Current project that i'm working has a requirement to capture data relevant to a session and store. Purpose of this is to use these data for the analytics. Clients would use these data to generate graphs, chart and etc.

For each client, 10 data points are captured within a minute(10 rows within a minute). As per the project requirements, table would exceed millions of rows within one week.

Current solution is to store the data in a Mysql database? Would this be the proper solutions?

  • well yes MySQL would be nice for this kind of data you could then pipe it to other services for further processing, but is it necessary to keep data for long such long time ? eventually you are processing those information and make something useful out of it as you produce whatever that is from data, there no reason for it to exist you could keep an archive in flat files or CSV MySQL engine just do you don't delete it and yet you can access when you need them. – Gntem Apr 24 '17 at 16:26
2

Databases can store many (hundreds of) millions of rows and even dozen of billions of them. You need just to create the appropriate database schema, and you should care about good indexing.

Of course you need enough disk space. This is the most important issue (and you need a careful estimate of the required disk capacity).

BTW (without knowing your actual requirements), by guessing a kilobyte per row, your database will certainly fit in a single terabyte disk (and more probably, only a few hundreds gigabytes of disk). You probably want the index to fit in RAM (so you might want a server with perhaps a few dozens of gigabytes)

You could use MySQL or PostGreSQL, but you could even use SQLite (see this answer). However, if you have multiple clients accessing (concurrently) the same database, you'll better use a real RDBMS server (e.g. PostGreSQL or MySQL), but not SQLite. If you want a lot (e.g. hundreds) of simultaneous connections, you may want to dimension appropriately your server (e.g. several disks, or SSDs to improve I/O bandwidth, or more RAM).

(I don't feel that your case involves a large database - I'll call it a medium-sized one, but I am not a DB expert; however, notice that PostGreSQL, MySQL and even SQLite all claim -see this for MySQL, this for PostGreSQL, this for SQLite- to be able to deal with multi-terabyte databases)

3

Redis is known as a cache server; and while it is great for that, Redis' real strength is as a session store. Redis is the clear choice for large and intensive session stores.

However, it is not clear to me that you are storing session data at all; but rather, it seems you are storing data about a session. There is a subtle, but important difference there. So, from your description, I think you could use a regular sql database without issues.

10 rows a minute per user is well within the capabilities of most mainstream relational databases. If you are really worried about performance, you can give each user their own database table, this will eliminate locking concerns.

  • 1
    Note : the module Spring session that replace old way of storing the session (typically HTTPSession with spring-web) use Redis as session store. – Walfrat Oct 9 '17 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.