0

I understand this is the place to ask questions pertaining to software design. If my question is considered to opinion-based, is there a forum or something better suited for the discussion?

Objective

My goal is to develop an application which can store & manipulate "historical data" covering the full range of time from the Big Bang (13.7 Ga) to the present day without loss of precision in the modern era. This encoding system should ideally represent dates as integers for the sake of fast queries. There is expected to be >100,000 records with dates spread throughout cosmological time but increasing in density towards the present day.

The will be a far larger number of records in the Holocene than in the several billion years preceding it. These records will also be dated to a higher precision; very recent events might have a time of day, requiring precision down to 1 second. Much older events might only be known to within a few thousand years, or more. I do not want to divide the timeline into sections! It is important than dates are handled in a consistent way across all of time, since any division would be arbitrary and un-physical.

Discussion/Solution

The existing UNIX epoch time is a good candidate to start from, already encoding timestamps to 1 second precision in a signed integer format. However, this scheme cannot handle very old dates given that the number of seconds since the Big Bang is approx. 4.3x10^17 which far exceeds the range of a 32-bit integer.

I see two potential solutions to this:

  1. Replace the 32-bit integer with a 64-bit integer, extending the range to roughly 1.8x10^19 (~50 times the required amount). The advantage of this method is that the encoding system remains linear and uniform. The disadvantage is in storage space and query time.
  2. Given the observation that precision is less important the farther back in time we go, we can consider the possibility of sticking with 32-bit integers, but using some kind of logarithmic scale instead. In this scheme, contiguous integers close to the modern age would represent a time difference of ~1 second, and this value would increase as we go back in time.

The signed range of a 32-bit integer is ~2.1 billion. Averaged over the entire history of the universe, this equates to roughly 6.3 years per integer, which is a promising result. If recent history skews this value towards ~1 second per integer, this can be balanced by skewing in the other direction at the opposite extreme. i.e. something like 100,000 years per integer in the vicinity of the Big Bang.

Thus, by taking advantage of the increasing uncertainty in dates as we move backwards in time, it is theoretically possible to encode all of cosmic history using a logarithmic 32-bit integer scale without losing precision in modern times... But is this a better approach? Do the performance benefits of 32-bit integers outweigh the cost of using a logarithmic scale?

  • Although dates like "formation of the earth" might be imprecise, things like end of the GUP epoc need sub nanosecond timing – Ewan May 1 at 11:21
  • I would say the correct way to do this is to divide up the date ranges into multiple calendars with varying degrees of precision. But you explicitly take this off the table. What are you actual requirements? – Ewan May 1 at 11:25
  • @Ewan Thanks for the response. You are correct that not all "very old" events have low precision dates. Well-defined epochs are such an example and this is another good reason why I might like to avoid using the logarithmic approach. The difficulty is that I want to be able to compare/sort dates quickly in a consistent way. I considered breaking the timeline into blocks, each with a range of 2.1 billion values, but this makes the query comparison more convoluted (I need to compare 2 numbers instead of one). – JeneralJames May 1 at 11:42
  • @Ewan I am also concerned that if I use multiple different "calendars" then this could lead to systematic errors such as trying to compare values which are encoded in different ways. The desire to have a single universal encoding stems from the need to compare dates quickly without having to run them through different transformations first. – JeneralJames May 1 at 11:45
  • 1
    what queries wouldnt be rapid? – Ewan May 1 at 11:54
3

I think if you have special needs for your date representation, you need to have a consistent set of rules where all the functions that receive a date know what to do with it. The whole idea that the Epoch should be January 1, 1970 is completely arbitrary. I don't think you can get away from that. However, we do have a problem with the current 32 bit date format in that we run out of numbers in 2038 (see the Year 2038 problem).

I think it's inevitable that we move to a 64 bit date. Microsoft used a 64-bit integer when they introduced the DateTime implementation in the .Net Framework. In that class, they also increased the precision so that it measures the number of "Ticks" since January 1, 1980. Yet another arbitrary epoch. A "Tick" is 100 nanoseconds or one ten-millionth of a second. I'm not saying that's the optimal solution, but it does work well in the .Net world.

With the cost of storage continuing to come down and the prevalence of 64 bit architecture, using a 64-bit date is feasible so long as all the functions know what to do with the number.

Having a 64-bit integer provides the following benefits:

  • Comparisons are extremely fast with 64 bit processors
  • There is no loss of precision between today and 10,000 years ago
  • Date math is simple to implement, which is partly why the Unix Date was made like it was

Of course, the obvious downsides include:

  • Requires twice the number of bytes to store
  • Working with external systems using a different epoch require you to do some math to convert

I would be nervous about using floating point values to represent dates as that subtle loss of precision can have unwanted side effects when two times are really close together.

  • The math to convert a 32 bit integer date to a 32 bit floating point date would arguably more complex.
  • Representing dates in textual format gets a whole lot more complicated as you lose precision
  • Simple date math like adding 7 * some_constant to find the same time next week is no longer simple as the multiplier is no longer constant
  • It would make bucketing more complicated so counts of records per month style queries take longer

At least this is my 10,000 foot poke at the problem.

  • Thanks! I was thinking about the year 2038 problem and coming to a similar conclusion. It's probably inevitable that we switch to a 64-bit scheme for timestamps anyway. I have more or less the same concerns regarding floating point numbers, it's really just a more error-prone method than using a 64-bit int. I just have this nagging feeling of wasted space for very old dates, since in the region of the Big Bang, you might only have a dozen timestamps across several million years. Perfect situation for a logarithmic scale, but it seems like the overhead is not worth the storage space it saves. – JeneralJames May 2 at 13:26
  • I'll mark this as the answer as it was quite helpful and I think more or less confirms the approach I was leaning towards. – JeneralJames May 2 at 13:28
  • 1
    @JeneralJames How tight are your space constraints? 100,000 64 bit dates only uses 400 KB more than the same number of 32 bit dates. Unless you're making this for an embedded system, or if you're often transferring all 100,000+ dates across a low-bandwidth connection, it seems unlikely you're really going to notice the size difference until you're in the 100 million record range and an extra 400 MB. – 8bittree May 2 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.