Cabling design of high-density optical fiber links in data centers

Today's data center equipment will be organized and divided into different functional areas: server area, storage equipment area, central switch area, router and high-performance cluster computer area. This orderly arrangement will be slightly helpful for the growth of service requirements, power supply and cooling system structured design.

As you can imagine, it is difficult to increase physical space, and the current structure can support this non-urgent demand at a relatively high cost. The data center will be divided into areas according to functional modules. When investment is needed for new demand, this structure will not hinder the effectiveness of investment due to the passage of time.

Promoting seamless connectivity

The interconnection problem on this infrastructure will affect the demand for line "interconnection" (or "series"). Long ago, the role of the data center was to connect to "anywhere". Many users in the enterprise or wide area network want to access all the services running on the server, so the server needs to directly access the storage devices, etc.

The core switch is to facilitate the connection of the backplane switching form, but it also means that the cabling needs to provide interconnection between each functional area, each module and each core area. The largest capacity and the longest cabling system have arrived, and as time goes by, more lines and faster link speeds will be needed.

The traditional solution, when there are not too many devices, is to use individual long jumpers to directly connect from the server to the switch and storage device. In a small data center, this is desirable. It has a relatively low cost and is feasible. But when the equipment and data center applications begin to grow, this point-to-point direct jumper method will slowly kill a data center.

Everyone has this experience. As time goes by, there will be more and more patch cords. The change and increase of equipment become more and more difficult to control and manage. Due to indirect storage loss, the performance and reliability of individual lines begin to decline. Finally, the available line laying space becomes crowded and blocked, which seriously affects the expansion of future applications. Today, the design of structured cabling is gradually being accepted by people.

It is well suited to the modular design of future data centers, increasing manageability, reliability and scalability. However, it also brings relatively high initial construction costs and requires attention to the additional losses caused by adding more connections in the wiring system.

When designing and building a structured optical cable system, several inevitable questions are: What kind of optical cable to use? How many standards will there be? For the question of optical cable, the answer will be relatively clear: OM3 (laser optimized 50um optical cable) can provide a relatively low total system price (optical cable plus laser transmitter), can support enough devices and link lengths, and the bandwidth can support the future 100Gbits/sec protocol.

The industry's OM4 standard has also been released. It is essentially a higher-bandwidth OM3 fiber that supports longer links and more applications. Because high-speed data applications often use shorter lengths of existing links, it is important to plan the structure of the line and how to upgrade it in the future.

The fiber quantity required requires detailed planning and calculation. First, consider the data switching structure and level of each equipment cabinet, and finally the level of each row cabinet, and the level of the collection. The second is to consider the type and density of equipment that will be used. However, there is no simple and easy answer. It is possible to design and provide the number of fibers in each cabinet based on the known type of equipment (such as blade servers).

There are two important points to consider:

1. Confirm the total number of optical fibers required for each row of cabinets, and configure the optical fiber usage of each cabinet that can be flexibly configured and changed in the future.

2. Plan the fiber optic line routing and interconnection fiber optic distribution boxes to facilitate the addition of fiber optic cables to each row of cabinets in the future. For example, when the processing capacity is upgraded from 10Gbit links to 40G and 100G, it means that each link is upgraded from using 2-core optical fibers to using 8 or 20-core parallel optical fibers.

According to the development of equipment technology, it may be difficult to provide an advanced number of optical fibers from the beginning. However, it is possible to use a flexible optical fiber wiring system to support future needs, and it will not have a big impact on the overall wiring system when the number of optical fibers is increased in the future. It is recommended to arrange at least 24 cores of optical fiber in each cabinet.

For applications such as virtual computing, high-density blade servers, centralized Ethernet Fibre Channel, 10G lines, etc., arranging 48-core or 96-core high-availability fiber distribution boxes in each cabinet is a good choice.

Dangerous jumpers

The most basic and necessary component, the patch cord, is the easiest to manage in a system, but it can also be the most complicated and troublesome. Common questions are: Where is the other end of this patch cord? What bad effects will occur if it is not plugged in? How or where can I provide new wiring?

A sound labeling and documentation system is the first priority, but most jumpers, especially in high-density application areas, are often beyond the control of user logic.

Port density will be limited by the type of fiber switch interface that fits the fiber optic switch. SFP fiber optic converters and duplex LC connectors have reduced the size of fiber optic connectors and increased fiber density by a factor of two in recent years. In contrast, small standardized connectors have reduced the size of cables from 3mm diameter to 2mm or even smaller.

But now, we have too many fiber patch cords, and they are already a bit tired because of the increasing problems caused by bending in the field. Fortunately, a new, flexible, bend-insensitive optical fiber has emerged and can solve the above problems well.

An additional method is to use Harness multi-fiber combination patch cord. It can concentrate the size of 6 patch cords into a 3mm cable. When Harness patch cords are used more, it can also reduce the size of the patch cords connected to the switch interface, making it better accessible and the cable direction is clear, and management is easier.

After using the industrial standard MPO connector, under the same volume of the dual-core LC connector, the harness patch cord at one end can greatly improve the density of the patch cord and reduce the space used for line interconnection in the cabinet. The next generation of parallel optical transmission rates of 40G and 100Gbit/sec will use this MPO structure laser light source interface.

The wiring box becomes more important

There are two basic requirements for fiber optic distribution boxes: providing key fiber functions and protecting interconnected fibers. Usually, fiber optic distribution boxes take up valuable cabinet space, so it is a good idea to occupy less cabinet space. However, too little space will affect the easy access and reliability of the fiber, or affect the organization and management of the fiber optic system.

So this metal box actually has an important meaning: when designing, more fiber connection points should be stored in it intelligently and less space should be used without affecting the management of jumpers and the function of the system.

A well-designed fiber optic terminal distribution box should have the following qualities:

A good mechanism for torque relief on each cable entering the distribution box. One of the problems is not using the spinning yarn pull line in the cable when pulling the fiber connector. A 144-core tight-buffered cable, if terminated in the field, needs to be about 0.92 inches in diameter, which will be twice the diameter of the factory pre-terminated system cable.

Make sure that the jumper can be easily inserted and removed with your fingers on the front of the panel, and make sure that the jumper is properly locked on the coupler. Also, when a jumper is removed, it will not affect other jumpers. There is nothing worse than pulling the wrong jumper or pulling the wrong jumper while it is running.

A clear, organized and easily visible labeling system should be right in front of you, not hidden away or somewhere you need to open. It should also be easy to add, delete and change the information on it. An outdated label is worse than no label at all.

Organization and management of jumpers. When the density of the optical fiber system increases, it becomes difficult to avoid the chaos of a large number of jumpers. A good patch panel will provide a direct view, routing guidance and easy management of jumper changes, which are all matters that technicians pay attention to.

Correct connection

The increasing importance of fiber optic connectors is also a problem. All connectors in data centers are now factory-made, whether it is fiber optic patch cords or terminated trunk cables. But the key to connector selection and its performance is to provide any connection. Duplex connectors are necessary in several connection modes (transmit and receive), cross-connects and equipment interfaces.

The most commonly used connector today, which provides small size and high density applications, is the two-core LC connector. In the interconnect area, those fibers begin to focus, and more fiber connectors provide higher density. The most popular one now is the 12-core MPO connector.

After the upgrade of 40G and 100G network protocols, we now call it parallel continuous optical path coordination, and PMO connectors are beginning to become more detailed and become future equipment standards. Another challenge is that the cabling system needs to be seamlessly upgraded from 10G to 40G and 100G, as if some applications will exist, but the connector type needs to be changed.

The good news is that 12-core MPO trunk cables have become very popular, and it is adaptable to future changes without changing connectors, just managing polarity and the fiber optic distribution box is ready for high-core-count cables.

In the overall loss plan of an optical link, the fiber connector is usually the weak link. As the network protocol gets faster and faster, the overall loss margin starts to become tight and begins to be controlled, and the loss of the connector has exceeded the loss value of the cable itself. In short, a pair of fiber connectors with lower loss will be more ideal. However, for multi-core MPO connectors, each fiber group is equally important, so the maximum loss is already considered.

A more serious concern and consideration is the transmission mode noise caused by the connector. If the effect of mode noise is not well understood, it will not be easy to explain that adding more mating connectors can increase the acceptable loss of the connector. This is a complex issue and difficult to test in the field. So make sure the connector supplier understands this issue and has done relevant quality evaluations in the laboratory.

Structure of the fiber optic system

Optical cables were first used in large quantities in outdoor factories. The cabling system depends largely on its needs. When optical cables began to migrate to indoor use, three important fiber optic cabling design directions were summarized:

1. The minimum required number of optical fibers has increased from 6 to 12.

2. Comply with the fire protection requirements of the building, which usually results in a large number of cables being required to have low smoke and flame retardant jackets.

3. There are a large number of short-distance cables and fiber optic connectors, so a connection method that can be used directly is needed.

Therefore, the thin optical fiber is made larger to facilitate on-site handling and termination. In the past few years, more optical links have been used indoors, especially in data center environments. As a result of the increasing number of fiber cores, the cables have become thicker, harder and more difficult to handle. So the end user needs to face a choice, should they use a cable with a larger number of cores or use several cables with smaller cores that are easier to handle?

Especially when there is a row of cabinets that are all core switches and need to be connected to the server cabinets, using a few small-core cables seems more attractive because it can connect the optical cables directly to each cabinet instead of the row cabinet. However, over time, this approach will lose the opportunity to easily and effectively manage the wiring of this row of cabinets.

So we need a solution that can achieve a high core count while having the same characteristics as thin cables. Due to the application of optical fiber in data centers, on-site pre-termination has become more attractive. In some large projects, the problem of tens of thousands of independent optical cable connectors in data centers is solved simply by using a plug-and-play system with on-site pre-termination.

When using factory-terminated fiber optic cables, you must calculate the length, which requires some margin in the calculated value because the cable needs to be coiled. Therefore, small cable diameter and flexibility become important. Small diameter cables can be stored in larger quantities in cable ducts, conduits, underground and ceilings.

They are lighter, more flexible, and have a tighter turning radius, making them easier to install. The combination of smaller, more flexible, factory-terminated high-fiber-count cables becomes a very useful and user-friendly system design tool, retaining the key design goal of factory-produced cables with high adaptability.

Using a good structured cabling manufacturer's products is a valuable asset in the data center.