PCIe 7.0 Interconnects — Is the End of PCIe Light?
Transmission Technology Challenges
PCIe technology has gone through Gen1-Gen6 and has developed to a single lane 128G transmission rate. The ultra-high-speed transmission rate has brought huge challenges. The association has to add more and more complex auxiliary mechanisms to control signal and data integrity. So, today we will mainly discuss the necessity and challenges of future PCIe 7.0 optical transmission technology.
Computing resource limitations
Facing the new polarized data center scenarios of super-large clusters and ultra-light edges, future computing will break through the bottleneck of the von Neumann architecture. Modules such as computing, storage, and communication will be interconnected through a unified bus. PCIe, as the main force for interconnection between data center servers, bears the heavy responsibility of high-speed data transmission. Computation-intensive tasks in data centers place higher demands on memory bandwidth and utilization. At present, most data centers still rely on local memory, which not only limits the speed of data processing, but also leads to inefficient utilization of memory resources. The optical transmission solution can achieve decentralized computing resources by enabling processing units to access more memory units distributed in different server units or racks (cross-server, cross-rack access), thereby releasing highly localized and often unused local memory, enabling data centers to allocate resources more efficiently.
The transmission distance of electrical signals is limited
The rapid iteration of large language models cannot be separated from the strong support of massive GPU clusters. At present, this massive amount has reached the level of thousands of cards, and the interconnection between clusters is usually based on the native PCIe interface on the GPU. As far as PCIe technology is concerned, when PCIe 1.0, the copper cable transmission distance is 10 meters, and when it develops to PCIe 5.0, this distance is shortened to 1-2 meters; when the rate is further increased to 64 GT/s and 128 GT/s, that is, PCIe 6.0 and the future PCIe 7.0, the copper cable transmission distance will be further shortened to tens of centimeters, and it is almost impossible to achieve the transmission requirements of tens of meters between racks through PCIe standard copper cables. In addition, PCIe technology needs to take into account the use of retimers, which are complex, expensive and power-consuming. And as PCIe technology is upgraded, the transmission distance is shortened, and the number of retimers required will increase, which will also introduce more power consumption and delay. Optical connections transmit signals through optical fibers, which can maintain the integrity and stability of signals at extremely high bandwidths. It can not only significantly increase the data transmission speed, but also reduce the delay of data transmission.
Cost measurement
As we mentioned earlier, the increase in bandwidth will lead to an increase in the number of retimers in the link. Under the same transmission distance, if optical transmission technology is used, fewer retimers and SCUs (signal conditioning units) will be required. In addition, the design and manufacturing process of optical components is relatively mature, and the cost will become more controllable. In addition, the space occupied by optical fiber is significantly smaller than that of copper cable, which also has the opportunity to increase the overall density of the data center and reduce system costs.
On the other hand, the transmission rate of a single Lane 128G will directly challenge the PCIe wiring level. The Gen5/Gen6 wiring standard provides the option of using copper cables to transmit PCIe within and between systems. Copper cables have less signal loss than PCB traces and can overcome the direct disadvantages of high-frequency communication. The technological upgrade will enable PCIe 7.0 to use thicker copper cables to overcome high-frequency communication, which will inevitably increase costs compared to the previous generation of technology. The PCIe 7.0 technology upgrade will also bring challenges to the corresponding server PCB process. With the soaring signal rate, the number of PCB product layers will increase, the BGA spacing will decrease, the board thickness will increase significantly, and the thickness-to-diameter ratio will increase significantly. These changes will also lead to a significant increase in material costs and processing difficulty.
As shown in the figure below, the theoretical loss and distribution defined in PCIe 7.0 Rev 0.5 version requires a pad-to-pad loss of -36dB at the Nyquist frequency point (32GHz) of 7.0. Compared with 6.0 and 5.0, the loss at the same frequency point (16GHz) is greatly tightened.
The feasibility of optical transmission technology
PCI-SIG announced the formation of the PCIe Optical Working Group in August last year, and plans to adopt a variety of technologies to support PCIe, including pluggable optical transceivers, onboard optical devices, co-packaged optical devices, and optical I/O. This year, we have also seen manufacturers from different industry chains conducting research on optical products based on PCIe. A more practical example is that a server manufacturer has extended the signal transmission distance from 1.4 meters to 20 meters based on the optical interconnect solution of PCIe Gen5. This solution successfully demonstrates the superiority of optical transmission.
However, it is difficult to achieve the transition of optical transmission technology in the short term. The initial PCIe interface did not consider the possibility of optical transmission, which means that the existing PCIe technology architecture is almost based on electrical signal transmission design. First, we need to consider the compatibility of optical transmission with the electrical layer, followed by the adaptability with the PCIe protocol layer, and then how to transmit PCIe signals through optical fiber and how to formulate the optical fiber-based PCIe form factor standard, how to formulate the FEC standard, etc. The editor believes that with the improvement of the association's technology and the participation of more and more manufacturers, we will be able to see a clearer prospect for the application of PCIe optical transmission technology.
At the exhibition at the beginning of the year, Keysight Technologies also jointly exhibited the pre-research of the test solution based on Gen7 with various manufacturers. Next, let's take a look at the details of the Gen7 test solution based on optoelectronic technology.
Traditional electrical layer solutions
Keysight Technologies and ALPHAWAVE SEMI jointly demonstrated the 128G Gen7 transceiver solution at the DesignCon 2024 exhibition held from January 30 to February 1, 2024.