Data acquisition, storage and management

Question

I have a device that can measure different values over time, one sample per second. After one measurement run I can export the data in form of a CSV-file. One row per second with timestamp and about 20 columns (values).

The question now is how to store these data masses to make the data accessable. I want to work with it using Matlab. I have for example 50 files of data, each representing one measurement run. Each containing around 120 tsd. samples of 20 values of type float, for example a temperature.

It would be great if I could request all samples at the same temperature for example.

So actually I am looking for a software that can handle and organize that data or a hint how (and what kind of) database to build.

Thank you for your help!

Answer 1

If you know the exact questions you want to ask from data and response time is not a big deal, you can use a program (written in any language of your choice) to go through the data prepare summaries every given interval so that the data volume becomes manageable, then load the summaries into a free database like MySQL where you could use SQL to get the results or you could load the summaries into Excel if the volume is small.

Short of that, if you need to store every data record without aggregation or summaries, you will need to have a process to load the data (possibly in batches based on time) into a database (SQL Server, Oracle, MySql, FoxPro, etc.). The process might be built via a database utility or an ETL tool.

The choice depends on how much money you want to spend, and on the level of expertise you have with the chosen tool.

There are applications that can process very large data in memory and produce good statistical information. OLAP tools can do some or all of the above in addition to data presentation. (see for example:(Link-1 ) or (R-Link).

The processes involved in satisfying your requirement are covered in the areas of ETL, OLAP, Data Marts and Data Warehousing as well as Dashboards.

Answer 2

MRTG may offer something for you if you are bothered about generic behaviour and not specific behaviour as time goes backwards. Although it is a way to consolidate data and not to store it precisely.

If you have only specific question you know ahead of time, then you can pre-parse your data. If not, you need to either have some database or load the whole data in memory and deal with it that way.