IXIA Data Center Ethernet (DCE) Test Solution

1 Introduction

As data centers expand in size and networking becomes more complex, the traffic generated by a variety of business applications makes management technology more complex and costly. With the recent development of Ethernet rates (10Gbps is already a mature standard and has been widely used, 40/100Gbps is still under development, and the official standard is scheduled to be launched in 2010. Ixia, a US company, has already launched 100G test instruments.), Ethernet has become the most attractive technology for the integration of next-generation data center storage and data transmission technologies.

With the development of data center technology, the next generation of storage networks (SANs) must integrate data center networks. Fibre Channel (FC) is a mature and reliable technology and has been widely used in SANs networks. Ethernet technology is the theoretical and de facto standard for local area networks (LANs). The currently deployed Fibre Channel technology has a maximum rate of 4Gbps, and it is unlikely to support 8Gbps soon. Ethernet already supports 10Gbps, and the 40/100Gbps standard is also being rapidly promoted. In order to meet the needs of server clusters and network convergence, a clear and definite direction is to deploy full Ethernet technology and support interoperability and backward compatibility. Fibre Channel over Ethernet (FCoE) technology is to directly transmit fiber protocols through Ethernet, so that data in the storage network can easily cross the boundaries of fiber and Ethernet, thereby reducing the cost and complexity of users in the construction and management of storage networks. The concept of Data Center Ethernet (DCE) was thus proposed.

2 Introduction to Data Center Ethernet Technology

Data Center Ethernet (DCE) is an enhancement of the current Ethernet technology standard in terms of high performance, and can provide a network architecture for the next generation of data centers. It should be noted that Data Center Ethernet has made a key expansion of Ethernet in terms of network structure scalability, operability, and transmission flexibility, with the goal of achieving a stable and lossless transmission layer.

In order to meet the above requirements, some new technical features such as mapping from Fibre Channel to Ethernet, "lossless" Ethernet and redundant paths and failover must be improved and enhanced to meet the needs of data center technologies such as FCoE. The main enhanced technologies are shown in Table 1.

2.1 Mapping from Fibre Channel FC to Ethernet

From the perspective of the OSI model, to map from Fibre Channel to Ethernet, the IEEE802.3 layer and FCoE mapping layer in Ethernet are used to replace the FC-0 and FC-1 layers in the Fibre Channel protocol model. Other layers remain unchanged, as shown in Figure 1. It should also be noted that the maximum length of an Ethernet frame is 1518 bytes. A typical Fibre Channel frame is 2112 bytes. As shown in Figure 2, since FC frames are larger than Ethernet frames, fiber frames need to be sent in segments when they are packaged on Ethernet and then reassembled at the receiving end. This will result in more processing overhead and hinder the smoothness of FCoE end-to-end transmission. From the perspective of the data frame type, FC and FCP protocols will be encapsulated in Ethernet frames and marked with Ethernet type 0x8906. The FIP protocol is similar to FC and will also be encapsulated in Ethernet frames, but the Ethernet frame type is marked with 0x8914.

To solve the difference between the transmission frame sizes of Ethernet and Fibre Channel, a larger Ethernet frame is required to balance the difference in Fibre Channel and Ethernet frame sizes. There is a de facto standard called "jumbo frames", which, although not a formal IEEE standard, allows Ethernet frames to reach 9k bytes in length. When using "jumbo frames", it should be noted that all Ethernet switches and terminal devices must support a common "jumbo frame" format. Because under normal circumstances, Ethernet switches will discard jumbo frames.

The largest jumbo frame (9K bytes) can encapsulate four Fibre Channel frames in one Ethernet frame. However, this makes the Fibre Channel link layer recovery and the buffer flow control of the 802.3x pause command more complicated. As shown in Figure 2, a complete Fibre Channel frame is encapsulated in a jumbo Ethernet frame, and the header information is 12 bytes (6 for the destination MAC address and the source MAC address), but the MAC address is storage transparent and can only be used to exchange frames from the source to the destination. Since the Fibre Channel addressing required for storage transactions is retained in the FCoE frame, a method for mapping from FCID (FibreChannelID) to Ethernet MAC address is required. A protocol similar to the Address Resolution Protocol (ARP) can be selected to implement the address mapping from FCID to MAC. For example, in the third-layer IP environment, the Address Resolution Protocol is used to map from the upper-layer IP network address to the second-layer hardware MAC address. The FCoE header information is 2 bytes, including control information such as version, SOF, EOF, etc. In addition, Fibre Channel uses some well-known addresses to obtain storage services (for example, through the SNS device discovery mechanism). FCoE requires the corresponding functionality to complete the mapping from well-known addresses to corresponding MAC addresses.

In traditional Fibre Channel, HBA or storage port receives FCID when connected to Ethernet switch. FCoE equipment cannot ensure that general Ethernet switch provides specialized storage services, so it must rely on domain controller and storage service engine available inside FCoE switch to provide Fibre Channel login, addressing and other advanced services. The need for these technologies provides guarantee for the integration of Ethernet, Fibre Channel and FCoE storage services.

2.2 Lossless Ethernet

The first challenge encountered in the development of FCoE is to continue the flow control mechanism implemented by the Buffer-to-bufferCredits feature of the native Fibre Channel. Although Ethernet switches do not have a corresponding buffer-to-buffer mechanism, the Ethernet standard can regulate the amount of information flowing in by supporting MAC control frames. The IEEE802.3x flow control standard is based on the pause frame flow control technology. This technology will cause the sender to delay the subsequent transmission content for a specific period of time before sending it. If the receiving device clears the buffer before this period of time has passed, it will resend the pause frame and reset the end time to zero. This allows the sender to retransmit until another pause frame is received.

Because the FCoE mechanism must support the reading and writing of storage data, all terminal devices and Ethernet switches in the network storage path must support bidirectional IEEE802.3x flow control. Although the effect may not be as ideal as the Buffer-to-bufferCredits mechanism, the IEEE802.3x pause frame can provide corresponding functionality to regulate storage traffic and prevent frame loss caused by blocking and buffer overflow. For storage transactions, this helps to enhance the service level quality of the flow control mechanism, so that the most critical task data flow can get the highest priority in the case of possible blocking. It should be pointed out that the commonly used 802.3x flow control on Ethernet can "pause" both control packets and data packets, but the 802.1Qbb based flow control can forward specific priority data flows. Figure 3 shows the impact of enabling flow control in Ethernet and 802.1Qbb flow control on FCoE traffic when running the Ixia IxExplorer FCoE test function. [page]

2.3 Redundant Paths and Failover

The high availability feature of Fibre Channel is mainly due to the Flat or CORE/EDGE topology network that can provide redundant paths between hosts and target devices. A failure at any point from the host bus adapter, link, switch port, switch or storage port from the primary path to the secondary path will cause the failure of the entire network. In some cases, both paths are dynamic and have both high performance and availability. The fiber shortest path first protocol in the Fibre Channel architecture is used to determine the best path for transmission between fiber switches, and its judgment is based on the link bandwidth and traffic load of the switch.

The Ethernet infrastructure must provide FCoE with the appropriate obstacle tolerance to ensure unimpeded storage access. When multiple Ethernet switches are connected through intra-switch links (for example, in a complete network topology), the IEEE802.1D Rapid Spanning Tree Protocol establishes a primary path on the network to prevent the transmission of frames from forming an endless loop. The dynamic bridge ports between switches are in a push state, and the non-dynamic failover bridge ports are in a blocking state. However, since blocked connections cannot be used for data transmission, blocked connections in the network represent unused and idle resources. Rapid Spanning Tree monitors the status of all bridge ports through bridge protocol data units. If the connection, bridge port, or switch fails, the Rapid Spanning Tree Protocol starts the necessary failover bridge ports to establish a selection path on the network.

In addition, IEEE802.1s Multiple Spanning Tree Protocol (MSTP) and IEEE 802.1Q-2003 Virtual LAN (VLAN) technology define additional mechanisms for enhanced Ethernet path switching. Similar to the hard zoning technology of Fibre Channel, VLAN tags can enable up to 4096 cluster node groups to coexist in a common Ethernet infrastructure. The enhancement of spanning tree on the multi-service transmission platform can enable a separate spanning tree in each VLAN group. Therefore, a bridge port in blocking mode of a virtual LAN can be adjusted to the forwarding mode of another virtual LAN, and achieve a fuller utilization of all network interconnectivity.

Even with the enhancement of multi-service transport equipment, the existing network connections still inevitably lead to the dependence of the Rapid Spanning Tree Protocol on forwarding and blocking states. Increasingly complex Layer 3 routing protocols, such as Open Shortest Path First (OSPF), select the best path between end nodes based on hop count, bandwidth, latency and other metrics, and implement load balancing on multiple paths. RSTP, as a Layer 2 protocol, cannot support such additional functionality while maintaining backward compatibility. It is necessary to find ways to introduce load balancing, multi-point access (for example, a node has two dynamic links to the same Ethernet segment), multicast technology, and broadcast technology to Layer 2 Ethernet.

3IXIA Data Center Ethernet Test Solution

Ixia, a US company, has developed a complete test solution that can be used to test the function, performance, security and scalability of FCoE devices. Figure 4 shows the test of Ixia emulating the FCoE initiator and target in a data center network.

Features currently supported by Ixia’s data center Ethernet solutions include:

Support FCoE features

Support FCoE Initialization Protocol (FIP)

Support lossless Ethernet features

FCoE and FIP tests are implemented using protocol and traffic wizards for easy operation

FCoE virtualization: ENode and VN_Port simulation, as shown in Figure 4

Support FCoE initialization and maintenance protocol

Support FIP ​​discovery, initialization and maintenance protocols

Support name server registration

Support for custom name server configuration sets

Supports full Fibre Channel target and initiator

Support stateful implementation of SCSI commands over FCoE

Supports 8 groups of priority flow control (PFC), as shown in Figure 5

Supports priority group-based statistics for PFC performance verification

Automated test suite for FCoE performance testing and latency analysis to facilitate performance testing

Supports running traditional Ethernet technology and FCoE technology on the same port

Each port achieves line-speed 10Gbps traffic, and a rack-mounted chassis supports up to 96 ports

The standards supported by Ixia data center solutions are shown in Table 3.

Using the Ixia FCoE solution, priority flow control PFC, ETS, congestion notification (CN) and data center bridging switching protocol (DCBX) can be simulated on an Ixia test port. At the same time, the Ixia port can also generate various types of traffic running on FCoE to simulate real network conditions and environments.

4 Conclusion

In November 2008, Ixia, as the only FCoE instrument provider, participated in the FCoE technology demonstration organized by the Ethernet Alliance in cooperation with several leading FCoE solution providers in the industry. The unique features of FCoE provided by Ixia include:

The industry's only true FCP (SCSI) traffic feature running on FCoE;

Mature and stable platform and design structure, convenient for testing FCoE switches, CNAs, drives and SAN network architecture

Simulates real Fibre Channel devices

Can send data to real Fibre Channel targets

Can receive data from a real Fibre Channel initiator

These features of Ixia products provide reliable guarantees for the relevant tests of data center Ethernet technology. Ixia's data center Ethernet test solution will continue to maintain its leading position just like Ixia's traditional 2-7 layer test solution.