Facebook releases Time Card clock synchronization open source project for TAP applications

The Time Appliance Project (TAP) is Facebook's latest open source system for sub-microsecond network time synchronization, which aims to provide very accurate timing and time synchronization across data centers in a cost-effective manner.

Data centers require synchronized time between servers, databases, and services to facilitate a wide range of applications such as electronic transactions, voice and video, and wireless sensor networks.

Recently, Facebook released Time Card, an open source x86 PCIe interface card designed to provide precise synchronization to tens of nanoseconds. The Time Device project includes all the hardware on the Time Card.

Facebook's time card/time device solution. Image courtesy of Ahmad Byagowi

At the heart of the card's design is u-blox's RCB-F9T timing board. u-blox's ZED-F9T multi-band timing module is also considered a key technology in Facebook's latest hardware project.

Facebook launched TAP in partnership with Open Compute with the goal of fully open sourcing the project and solving many of the problems associated with proprietary time devices.

Learn more about Time Appliance hardware

The Time Appliance project is available as a GIT repository and includes hardware files, software files, bill of materials, and details about the Precision Time Protocol (PTP) profiles developed within OCP.

The time card itself, which forms part of the time device project interface, was built in Altium Designer. It is not a high-density design. However, as the design is intended to interface with relatively high-frequency GNSS systems, signal integrity and EMC data would be a welcome addition to the GIT file.

The PCB is a 62 mil thick six-layer board using two ground reference planes on layers 2 and 5. Layers 3 and 4 are orthogonally routed to avoid significant effects of broadside coupling.

Time card PCB layout (reference planes disabled) detailing the layout design and density. Image from Facebook design files

Building the Time Appliance Project

The Time Appliance is built using three main hardware components:

GNSS timing source (RCB-F9T board from u-blox)

Miniaturized Rubidium Atomic Clock (Microchip's MAC-SA5X)

System-on-Module FPGA (Alinx's Artix-7 AC7100B SoM)

Time Card prototype. Image courtesy of Facebook

The FPGA system receives two main signals from the GNSS: a pulse per second (PPS) reference signal and a time of day (ToD) signal. The system is clocked by a rubidium oscillator or an external 10 MHz.

After filtering and processing the two synchronization signals, the resulting reference signal is output to the PCI bridge to the x86 system network interface card (NIC).

High-level block diagram of the Time Appliance FPGA. Image courtesy of Facebook

Digging deeper into the FPGA subsystem, Facebook outlines the filtering of the GNSS input signal (DCM).

FPGA digital filtering subsystem. Image courtesy of Facebook

Ultimately fed into the ToD unit, these filtered signals represent the accumulated increments of 8 nanosecond digital values. The goal is to take a snapshot of the incremental digital counts using the GNSS signal to achieve an ideal 1 second PPS.

The system has built-in redundancy, allowing the MAC to take over for a period of time in the absence of GNSS signals; specifically, the incremental digital count should be maintained within 1 microsecond over a 24-hour period to achieve the defined time loss accuracy.

The card is designed to interface with any x86 host system that has a hardware timestamping capable NIC, turning it into a “time device.” Synchronization of the system is protocol agnostic, allowing use of the Network Time Protocol (NTP), Precision Time Protocol, or other alternative synchronization protocols.

Why did Facebook choose to open source TAP?

TAP is not the first open source project Facebook has released to the wider community. In 2011, Facebook publicly released the design of its state-of-the-art data center and launched the Open Compute Project Foundation with other industry members.

Facebook has recognized the difficulties of using proprietary systems: long development times to patch security holes, closed-system software with limited configurability, hardware that is not user-serviceable, and high costs.

The time card is a relatively cost-effective timing card that can be built using COTS components and PCB design, lowering the entry barrier to the sub-microsecond synchronization market.

All in all, as an open source project, there is much more information available to interested designers, including hardware, software, or available protocol stacks.