The application will continuously (approximately every second) collect the location of users and store them.
This data is structured. In a relational database, it would be stored as:
| user | timestamp | latitude | longitude |
However, there is too much data. There will be 60 × 60 × 24 = 86,400 records per user, daily. Even with 1000 users, this means 86,400,000 records daily.
And it is not only 86,400,000 records daily. Because these records will be processed and the processed versions of them will be stored as well. So, multiply that number with approximately 2.
How I plan to use the data
Essentially, I plan to make coarser grained versions of location data for easier consumption. That is:
- Sort the received data w.r.t timestamps.
- Iteating on this list in order, determine if the location has changed significantly (by checking out how much the latitude and longitude changed)
- Represent the non significant location changes as a single entry in the output (hence, output is a coarser grained version of the location data).
- Iterate this process on the output, by requiring an even larger latitude and longitude change for a significant change. Hence, the output to be produced from the previous output will be even more coarse grained.
- Iterate the whole process as much as needed.
- Aggregate a range of resolutions and send them to users. Also, store all resolutions of the data for later consumption.
What should I use to store this data? Should I use a relational database or a NoSQL solution? What other things should I consider when designing this application?