You need to do this based on what you actually see in your system. The two basic approaches are to do it once or to make your code adjust on the fly. The first is obviously a lot easier to do but somewhat less resilient.
Doing it once means logging the time difference for each pair of servers for a while and seeing what you get. This might take a while, since you want to see whether they're re-synchronising to NIST or similar at intervals. But even very early on you should get an idea of whether the time differences are consistent or not. Then hard-code in some reasonable value. The biggest jump is likely to be the correction one, but you may never see that actually happen so you'll be guessing "our site is quiet overnight but the skip sees to happen between 2am and 6am AEST" is likely).
The more resilient but more code version is to do the above but map your actual offset to the most recently observed values. If your timeout is a couple of hours and you see the other system has been drifting a second a month just build that in to your pre-expiry. It only gets complex when you're trying to guess the point at which the other system will jump a couple of seconds back due to a monthly clock correction.
I think the latter is overkill unless you're looking at jumps within a factor of 10 of the total expiry time. At that point you're going to want to throw away 20% of the total time, which is a 25% increase in OAuth traffic. So it might be worthwhile to write more code in an effort to reduce it. But even then, that might not be the biggest bandwidth saving you can get for your programming time.