3

In a normal application, a simple call to time.Now() in most languages will suffice for getting a timestamp, but what about in a web application spread over multiple servers? As I understand it time.Now() will use the current system time, but there's no reliable way of telling after the fact if that time was synchronised with the rest of the machines. By this point you're throwing data into persistent storage with inconsistent timestamps which is not such a problem for created/modified times when inserting into a database, but a bigger problem when tracking event timing).

The best way I see to mitigate this is to have all machines request a timestamp from one specific 'master' machine, but then I thought what about latency? What if the network is busy and the response is delayed? There's several flaws with this method but I can't think of a more reliable one other than using CPU and network resources synchronising the time every few seconds.

What is the accepted best practice/industry standard here?

  • 1
    The Google Spanner team has some publications and talks on that topic. – Patrick Sep 3 '15 at 11:41
7

The usual practice is to set the servers to all keep their time updated using NTP. There are limitations to the accuracy when using NTP time syncing which means that you should only rely on the time stamp to give a general idea of when events occurred, which is likely to be good enough for identifying the set of events you are interested in.

Timestamps are no good for determining the sequence of events, even on a single server at a precision level, as even with microsecond timing, each CPU core will process 1000's of assembler instructions during the same timestamp. Your server may have 40 such cores.

The best approach I know of, to keep track of what is happening in a distributed system is to use transaction ID's and log files. A single log file written sequentially allows you to determine event order on a single server. Anything that passes work between servers will need to appear in the logs on both servers recording the transfer of the work with a shared transaction ID.

  • A common method for this is using a Vector Clock – ctote Sep 3 '15 at 15:39
  • 1
    Vector clocks are good when you have a highly cohesive system and need to the know relative sequence of distributed events. They are used for eventual consistency in Riak, for example. However, they do not give timing for a timestamp. – Michael Shaw Sep 4 '15 at 8:17
  • Sure, that was the point. Vector Clocks are an example of what you eluded to in your last paragraph. Or at least, that was the intent. – ctote Sep 7 '15 at 2:41
  • 1
    Actually, I was thinking a simple GUID at that point, as a transaction ID, to allow you to identify all log entries that related to that one transaction. Vector clocks are useful for computational determination of a sequence of related actions to work out the likely sequence, but they are a very painful way for doing log analysis, as per the question – Michael Shaw Sep 7 '15 at 10:44
1

apart from having a time server that every timestamp is supposed to come from (with the massive performance overhead in network traffic that may generate)?
The usual practice is to either a) not bother or b) have procedures in place that ensure the clocks on all machines are kept in synch from a single central NTP server.

  • A is not an option for me unfortunately. I've had a look at Lamport timestamps but I'm not confident that's what I need. The search continues. – leylandski Sep 3 '15 at 11:59
  • @leylandski then go for 2), ensure through procedures to be followed by your network and systems admins that the clocks are kept in synch. – jwenting Sep 3 '15 at 12:14
  • II) is still susceptible to errors, though. There is no such that as two perfectly synchronized clocks. – MetaFight Sep 3 '15 at 12:23
  • @MetaFight they're likely synchronised to within less than the accuracy of the timestamps your software generates. – jwenting Sep 3 '15 at 12:24
  • Hm, I didn't know that :) – MetaFight Sep 3 '15 at 12:27
1

As mentioned in @jwenting's comment, NTP is basically the "accepted standard". But if latency is highly critical for you, maybe TICSync is a useful option. It achieves millisecond accuracy very quickly and is used in robotics systems where a high synchronisation precision, beyond what NTP can offer, is required.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.