I'm working on a project that generates massive amount of data and store that data in a
SQL database. I'm talking about like 1 thousand records per second. Next I've to push that data to a database. If I do the data generation and database push sequentially, it'll slow down the data generation which is not acceptable. I considered keeping a buffer and two parallel process- one will write data in the buffer, and other will read data from buffer and push to database. This solution might work but the problem is due to write latency in database, the buffer grows too large and eventually causes memory limit exception. Writing to a file and then updating the database from file doesn't seem to be a very good solution either. From software engineering perspective, what other solution I can consider.
Edit for Comments:
- Writing to file doesn't seem to be a good solutions as the file size will also go too big. I feel that's not how a tech giant will solve this kind of problems.
- The data is generated from some calculation. The calculation is time critical and we expect one calculation per second. So 1K records per second.