Many systems need "synchronised" time, from distributed scheduling to monitoring systems. Synchronised time can be quite a hard problem to solve for if you need sub-1 second accuracy.
The easiest way to achieve high accuracy is to measure start and end times from the same location, though that may introduce a Single Point Of Failure and/or can have an impact on scalability.
Example
Lets look at a simple latency measuring system to highlight why this is a problem, and how to handle the behaviour.
Alice measures the start time, and Bob measures the end time of the transaction.
The latency is measured as:
One-way latency = T1-T2
This assumes Alice and Bob have a shared concept of time. If Bob's time was 1 minute fast then the latency is inaccurate to up to 1 minute. In this case, the one-way latency is:
One-way latency = (T1+Alice time delta from Real Time) - (T2+Bob time delta from Real Time)
To some extent Alice and Bob have a shared concept of time, and we use systems like NTP to synchronise network systems. As always, there are limits to the accuracy if you look close enough.
NTP ensures device time is synced, commonly to at-best 100ms[1], unless you have a GPS source on your local LAN. The time between two machines can therefore be potentially up to 200ms apart:
One-way latency = (T1+100ms) - (T2+100ms) = T1 + T2 + 200ms
For many systems this works fine, but if you're looking to measure sub-1s across the network then the error margin is far larger than the required accuracy.
The same problem exists if you try and measure from a third party for example, collecting data by packet capture and post-processing it. If the packet capture processing mechanism sees data from multiple sources, then it's unlikely to be accurate enough to use to calculate latency.
Resolving the problem
One simple and effective way to resolve the problem is to measure both T1 and T2 from the same point.
Time taken for Alice to see the full round trip (RTT): T1-T2
One-way latency: (T1-T2)/2
For differential measurements (like a latency measurement), we don't need to care about how far away from Real Time our measurement is, only that it is consistent.
One-way latency = (T1+32ms) - (T2+32ms) = T1 + T2
The delta is the same at T1 and T2, thus our latency measurement is accurate.
Conclusion
As is often found, complexity is everywhere if you look close enough. Keeping things simple can be a great way to engineer yourself out of a hole, and avoid design and support nightmares in the process.
---
[1] NTP Accuracy
http://www.ntp.org/ntpfaq/NTP-s-algo.htm#Q-ACCURATE-CLOCK
$ ntpstat
synchronised to NTP server (1.2.3.4) at stratum 3
time correct to within 188 ms
polling server every 1024 s
