High-Speed ​​Connector Power Design Challenges in Cloud Technology

summary

With the addition of cloud services (e.g. Apple iCloud) to the already widespread Internet, pure computing and storage requirements in data centers are growing at an unprecedented rate. This growth has a direct impact on energy consumption. As it continues to grow, engineers are looking for solutions to control power consumption. As the connectors used in networking systems transmit at speeds exceeding 10Gb per second (Gbps), this article will specifically look at connector power budgets and solutions to reduce the power consumption of these high-speed channels.

introduction

Today, there is no doubt that Internet data traffic is growing rapidly. This trend is clearly shown in the latest (June 2011) Visual Networking Index (VNI) forecast report from Cisco, which focuses on the growth of the mobile space (see Figure 1 ). The introduction of "cloud" computing and storage has driven bandwidth consumption to an ever-increasing level, driven by this novel paradigm. Mobile users are moving from simple text data transfers to high-definition photo and video transfers, with the urge to copy these contents to cloud storage, transcode videos for distribution, and copy multimedia data to a wide variety of devices (not to mention posting on social networks). This performance pressure ultimately requires increased processing and communication capabilities.

Figure 1 Internet bandwidth trends from 2010 to 2015

However, these increases come at a cost—not just in terms of money, but also in terms of power consumption. Designers of next-generation servers and networks are already struggling with power consumption—both in terms of cost of ownership (CoO) and practical thermal design. How can systems be built to increase performance while reducing power? In an age of explosive information growth, it’s a never-ending battle.

What to study first

As with all system designs, the next generation of servers and networks should first improve performance. In cloud computing architectures, services often change as load changes. It is no longer a "server", but a discrete hardware device. In most cases, the actual hardware that provides the service can be placed anywhere within the service provider's infrastructure, and its location is "uncertain" and can change at any time. This performance improvement is called "virtualization" or encapsulation of services within a software framework, which allows services to move freely between hardware hosts. This allows service providers to change resources based on demand, thereby reducing the power consumption of the infrastructure.

As services are controlled, there is a lot of "machine-to-machine" (M2M) activity. In most data centers, most data transfers occur between machines, without connecting to the outside world. The addition of virtualization drove the need to move from 1 Gbps connections (the standard on many older servers) to 10 Gbps connections. Today, this demand is driving the move to 25 Gbps connections. Many of these wires are less than 5 meters long, and most are less than 1 meter long. The reason for this is the architecture of server clusters. A single rack holds many blade servers, which are connected to switches at the top (or bottom) of the rack. The racks are placed in rows, which are consolidated by hubs, which then send information to other rows of servers or network storage.

When using 1Gb connections, small gauge cables can easily transmit data with minimal loss of signal integrity. This is important because 1) airflow out of the server is reduced because the cables block airflow outward, and 2) the bend radius determines how many cables you can run in the rack (see Figure 2 ).

Figure 2 Internal wiring of the rack

With the move to 10G Ethernet, signal integrity issues became more prominent and passive cables began to use larger gauge wires to compensate. Airflow/bend radius issues began to emerge and installers/designers began to look to fiber optic connections to address this issue. The move to fiber optics brought some issues such as high cost and high power consumption. A typical single 10G Ethernet SFP+ module consumes about 1 watt of power. When using tens of thousands of ports, the power consumption required for fiber optic connections increases dramatically and some of the problems caused by increased power consumption also arise (increased rack temperatures).

Cable connection problem

If passive cables for high-speed connections suffer from bulk and bend radius issues, fiber-optic solutions suffer from high power consumption and high cost. It seemed as if a compromise had to be found to solve this problem. The answer is a technology called "active copper wire"—a clever idea that embeds some active components into the conductor housing to compensate for the high-frequency losses caused by small-gauge wire. This solution allows the use of small-gauge wire with "fiber-type" bend radius and large size, but with higher power consumption. Devices such as the DS100BR111 typically consume less than 65 mW per channel at 10 Gbps and are commonly used in SFP+ active wire applications.

When applied to 10 Gbps Ethernet, this technique for improving cable signal integrity is limited to cable lengths under 15 meters in most cases. However, as mentioned earlier, most cables are under 3 meters, and passive or fiber optic cables can be easily replaced with active copper cables. Today, this approach is commonly used for 10 Gbps connections. However, the future is approaching quickly, and even 10 Gbps connections will not be able to meet the needs.

In the world of fiber optic connections, there are basically two types of connections: 1) short-distance connections (less than 1000 meters); and 2) long-distance (greater than 1000 meters) communications. Longer fiber optic connections form the backbone of our modern Internet infrastructure, often using 100 Gbps WDM fiber technology. In order to reduce the cost of this technology, major companies including Google, Brocade Communications, JDSU, etc. approved a 10 x 10 Gbps multi-source agreement (MSA) in March 2011 for the physical medium dependent (PMD) sublayer, which provides a common architecture for C form factor (CFP) modules.

The CFP connector is suitable for low-count/long-distance connections that require 100 Gbps communication. However, SFP and Quad SPF Interface (QSFP) connectors have higher density, which is required by local switches and routers. Today, quad-channel SFP connectors are used for 40 Gbps Ethernet by combining four 10 Gbps data channels. The next step in the evolution will be to move from 10 Gbps to 25 Gbps channels. It provides data transmission equivalent to 100 Gbps through some small QSFP connectors and provides a backward compatible mode for some 40 Gbps Ethernet systems that do not support the 100 Gbps standard. Eventually, this form factor can be used for fiber modules because the 10-to-4 channel conversion used by CFP modules is no longer required.

This technology has been proven many times by several vendors, providing a roadmap for infrastructure designers to move to high-speed connections. However, the interconnect behind a switch or server is not the only place where this problem occurs. The same problem exists in various electrical connections inside servers and network storage devices.

Distance is your enemy

The waveform of a digital bit traverses transmission lines and connectors, so physics kicks in and tries to completely destroy the original signal through frequency-reflection-type variable attenuation caused by impedance mismatches and adjacent channel crosstalk. The data itself is also problematic, as previously sent symbols interfere with the current bit in transit. This is called inter-symbol interference, or ISI. Once the signal has traveled the distance from the ASIC to the router or back of the switch, the bits can no longer be discerned. The same effects that kill error-free bit transmission over passive wiring are at work here, too.

In some previous designs, the switch ASIC used multiple slow data paths (typically 3.125 Gbps) connected to a physical layer device (PHY) to build a 10 Gbps NRZ connection at the SFP connector. The PHY was located very close to the physical connector, so signal integrity loss was minimized. However, as ASIC technology moved to smaller geometries, accommodating the high-speed connection of the 10 Gbps interface became an inherent requirement. First, this change can reduce the overall power consumption of the electrical connection by removing the PHY. However, signal integrity loss at the edge of the PCB requires more expensive, low-power board materials, or the use of an active solution.

The same devices used to combat cable signal loss are now being used in high-performance routers, switches, and server internal connections. When using low-power buffer interrupters and retimers, standard FR-4 PCB materials can be used (controlling costs) and power consumption is very low. In fact, these devices are used in a similar manner in 10 Gbps NRZ Ethernet PHYs to recover and retime data to meet connector specifications.

Efforts to achieve the target

In servers, standards such as PCI express (PCIe) are ubiquitous. Standards such as PCIe continue to increase in speed due to higher data rates and the ability of the core processor to transfer information to and from the core. The latest standard is Gen 3, which has a nominal connection speed of 8 Gbps. As mentioned earlier, in many cases, the physical distance within the device remains unchanged due to the processor hardware, the number of connectors, and the spacing. Servers are no exception and are also plagued by signal integrity issues and power consumption. Some previous designs using PCIe Gen 1 or Gen 2 were able to meet operating specifications with careful layout and connector selection. However, as servers move to Gen 3, board materials and connectors are affecting signal integrity to the point where this standard can no longer be met.

Standards such as PCIe introduce another problem that makes it more difficult to solve while keeping power consumption low. This problem is out-of-band (OoB) signaling, which occurs early in the channel training process. Since the standard PCIe board does not know when the channel is connected, it must communicate with the root component and make adjustments to the channel to help maintain signal integrity. This communication is done out of band, and if it fails (because it is blocked for some reason), the channel cannot initialize.

Some PCIe integrated circuit (IC) interrupt manufacturers use a repeated root component approach. This approach splits the channel in two, effectively shortening the distance and greatly improving signal integrity (fewer connectors/shorter distance). The problem with this approach is power consumption. Repeating the root component requires understanding the channel transmission and correctly repeating it at both ends. In addition, the serialization and deserialization process will cause excessive delays.

Other vendors have temporarily addressed this problem by using an analog approach to condition both in-band and out-of-band signals (removing all information processing). Devices such as the DS80PCI402 use this approach and require only 65 mW per channel. When inserted into a PCIe channel, the device effectively shortens the channel distance between the end node and the root component, does not interfere with out-of-band processes, and greatly improves 8 Gbps data signal integrity while consuming less power.

Other improvements

Our information infrastructure is growing to meet the needs of the increasing number of users and technologies (such as cloud computing, etc.). The connection power budget is only part of the total power consumption of these systems. Major manufacturers are looking for a way to produce cores with lower connection power consumption. Due to the ease of use and extremely low power consumption of ARM cores, people are paying more and more attention to using this engine in cloud servers. In addition, some special-purpose processors also use their own methods to enter the information infrastructure to provide various services, such as real-time transcoding of video and images, voice recognition, etc. These special-purpose services usually require floating-point operations to be performed in general-purpose processors. These special-purpose processors provide many energy-efficient methods to perform the same computing functions.

in conclusion

As cloud computing and storage continue to grow in both scale and capacity, the connectivity between nodes continues to increase. Designers are challenged to maintain the lowest power while increasing network data throughput. These solutions are not only challenged by the growing high bandwidth requirements, but also reach the upper limit of minimizing power consumption.

References

To learn more about signal conditioning, visit www.ti.com/sigcon-ca.

About the Author

Richard Zarr is a TI technical expert, focusing on high-speed signal and data path technology. He has more than 30 years of practical engineering experience and has published numerous papers and articles around the world. Richard is a member of the IEEE, graduated from the University of South Florida with a bachelor's degree in electrical engineering, and holds several patents in LED lighting and encryption.