Toward Trusted Time: Remote Server Vetting and the Misfiring Heart of Internet Timing

The core of the Internet's timekeeping system are the Stratum-1 timeservers, those connected to reference hardware, that anchor the server hierarchy. It is essential that these root servers are accurate and reliable, and this it is typically taken as a given. We examine this premise through an examination of 102 prominent Stratum-1 servers, using 3 datasets spanning 6 years, collected in reference testbeds with authoritative timestamping. We describe a methodology capable of rigorously removing congestion related variability, allowing server errors to be unambiguously revealed. We use the data and methodology to assess the health of public network timing, and how it varies over time, by reporting on the type, severity, duration, and prevalence of server errors, and how they relate to protocol level information. We present conclusive evidence that the system has problems. We find that errors are widespread, significant, often endemic, consistent over time, and typically come with no warning at the protocol level. Our results highlight the lack of oversight in the current system, and provides the foundation of a server health monitoring capability, necessary to restore and maintain trust in network timing. We describe three specific applications where our results can have an impact. Our data, detailed results and software are publically available.