I understand this is the place to ask questions pertaining to software design. If my question is considered to opinion-based, is there a forum or something better suited for the discussion?
My goal is to develop an application which can store & manipulate "historical data" covering the full range of time from the Big Bang (13.7 Ga) to the present day without loss of precision in the modern era. This encoding system should ideally represent dates as integers for the sake of fast queries. There is expected to be >100,000 records with dates spread throughout cosmological time but increasing in density towards the present day.
The will be a far larger number of records in the Holocene than in the several billion years preceding it. These records will also be dated to a higher precision; very recent events might have a time of day, requiring precision down to 1 second. Much older events might only be known to within a few thousand years, or more. I do not want to divide the timeline into sections! It is important than dates are handled in a consistent way across all of time, since any division would be arbitrary and un-physical.
The existing UNIX epoch time is a good candidate to start from, already encoding timestamps to 1 second precision in a signed integer format. However, this scheme cannot handle very old dates given that the number of seconds since the Big Bang is approx. 4.3x10^17 which far exceeds the range of a 32-bit integer.
I see two potential solutions to this:
- Replace the 32-bit integer with a 64-bit integer, extending the range to roughly 1.8x10^19 (~50 times the required amount). The advantage of this method is that the encoding system remains linear and uniform. The disadvantage is in storage space and query time.
- Given the observation that precision is less important the farther back in time we go, we can consider the possibility of sticking with 32-bit integers, but using some kind of logarithmic scale instead. In this scheme, contiguous integers close to the modern age would represent a time difference of ~1 second, and this value would increase as we go back in time.
The signed range of a 32-bit integer is ~2.1 billion. Averaged over the entire history of the universe, this equates to roughly 6.3 years per integer, which is a promising result. If recent history skews this value towards ~1 second per integer, this can be balanced by skewing in the other direction at the opposite extreme. i.e. something like 100,000 years per integer in the vicinity of the Big Bang.
Thus, by taking advantage of the increasing uncertainty in dates as we move backwards in time, it is theoretically possible to encode all of cosmic history using a logarithmic 32-bit integer scale without losing precision in modern times... But is this a better approach? Do the performance benefits of 32-bit integers outweigh the cost of using a logarithmic scale?