Design and implementation of multi-FC daughter card test equipment based on PCIE

 0 Preface

FC (Fiber Channel, the abbreviation of Fiber Channel) is a high-speed serial transmission protocol proposed by the X3T11 working group of the American National Standards Institute (ANSI) in 1988. It has the characteristics of high bandwidth, high real-time performance, high reliability, good scalability, high transmission rate, strong anti-interference, flexible topology and service type, and support for multiple upper layer protocols and underlying transmission media. In order to adapt to the application of avionics environment, Fiber Channel provides a set of protocol subsets used in avionics environment FC-AE (fiber channel avionics environment), which standardizes the selection of civil and military aviation fiber channel switching networks and ring topology connection devices. FC-AE is based on the upper layer mapping FC-AE-ASM of anonymous signed messages (ASM), which has the characteristics of secure message transmission and low latency, and is suitable for communication between processors and sensors and displays in avionics. Therefore, FC has become the preferred solution for the interconnection of the new generation of advanced integrated avionics networks.

In order to ensure that FC products can meet the requirements of airborne applications, it is necessary to provide a test and experiment platform for FC products and design reliable and efficient test equipment for airborne FC equipment. FC daughter card is a FC product widely used in avionics systems. FC daughter card realizes the function of FC node machine, is responsible for providing FC communication connection and other functions for each node in the FC network, and provides communication support for various application data.

According to the characteristics of the device under test, this paper proposes a design method for a test device with multiple FC daughter cards based on the PCIE interface. The PCIE interface of the PowerPC processor is adopted to expand the multi-channel PCIE bus through PCIE switching, so that multiple FC daughter cards can be tested at one time. It has been verified that the communication of the test equipment is stable and reliable, and it has high practicality, versatility and scalability.

1 Overall structure of multi-FC daughter card test equipment

1.1 Analysis of the measured object

The FC daughter card is an embedded FC node machine. As the interface for the avionics system to access the FC network, it has an interface for information exchange with avionics system equipment, namely the FC interface, which supports FC-PI, Fc-FS and FC-AE-AsM protocols, and the FC interface link rate is 2.125Gbps. On the other hand, it has an interface for data exchange with the processor, so that data on the network can be submitted to the application, and application data can also be sent to the network. Therefore, the above two aspects are mainly considered when designing test equipment.

Considering the requirements of structure and rate, the FC daughter card adopts X4PCIE host interface, with an operating frequency of 2.5Gb/s and a standard XMC structure. It consists of FGPA, configuration circuit, RS232 interface, FLASH interface, clock circuit, reset circuit, power supply circuit, and FPGA is used to implement FC interface, PCIE interface and on-chip processor. The hardware block diagram is shown in Figure 1. The FC daughter card mainly implements FC-AE-ASM communication, time synchronization, network management and other functions to ensure the integrity and reliability of avionics data transmission.

1.2 Test equipment design

The following aspects need to be considered when designing multi-FC daughter card test equipment:

(1) Meet the requirements of product function and performance;

(2) Consider using it in harsh test environments such as temperature shock, humidity and heat;

(3) Consider testing as many devices as possible in one test to improve resource utilization and test efficiency.

Since the main external interfaces of the FC daughter card are the PCIE interface and the FC interface, the device must be accessed and tested through the above two interfaces during design. For the PCIE interface, a PowerPC processor module with a PCIE interface is used, and the module needs to have necessary peripheral interfaces such as Ethernet, FLASH, SRAM, and serial port, so that software development and commissioning can be performed on it. At the same time, in order to test multiple FC daughter cards at a time, it is necessary to use PCIE switching to expand multiple PCIE buses. For the FC interface, the FC daughter card is implemented using FPGA to output FC electrical signals. In order to test a series of contents such as FC protocol compliance, functions and performance, connect external test equipment, and convert electrical signals into optical signals, it is necessary to design an adapter board, which connects the motherboard and the FC daughter card through the PCIE interface on the one hand, and converts FC electrical signals and optical signals through the photoelectric conversion circuit on the other hand.

In order to meet the requirements of high bandwidth, versatility and use in harsh test environments, a 3U VPX structure design is adopted, and the host and daughter card are connected through a motherboard and an adapter board. The overall framework of the test equipment is shown in Figure 1.

Figure 2 Overall framework of multi-FC daughter card test equipment

2 Key Technology Research

The key points of the design of multi-FC daughter card test equipment are PCIE switching and PCIE configuration. The following describes the selection of PCIE switching and the mechanism of PCIE configuration.

2.1 PCIE Switching

When choosing a PCIE switching chip, two points need to be considered. First, select the appropriate configuration based on the system requirements and the line width, rate, number and other requirements of the PCIE devices connected to the upstream and downstream ports. Secondly, choose whether the PCIE switch is a transparent bridge or a non-transparent bridge. Generally, a transparent bridge is used in a single-processor system, and a non-transparent bridge is used when connecting multiple processor systems.

2.2 PCIE bus configuration mechanism

PCIE is the same as PCI environment. Various devices are connected to the tree bus, including one or more functions. In this design, the FC daughter card is a single-function device. When the system is first started, only bus 0 in the root complex (RC) in the processor has a number, and other buses are yet to be discovered. The configuration software uses a depth-first search algorithm to scan all PCIE devices on the PCIE bus tree and allocate bus numbers and memory address space accordingly.

For RC or PCIE switching using Type1 configuration header, the primary bus (Primary Bus Number), secondary bus (Secondary Bus Number) and slave bus (Subordinate Bus Number) registers must be initialized. The primary bus is the bus upstream of the RC or bridge, the secondary bus is the bus downstream, and the slave bus is allocated from bottom to top and is the bus with the largest number in the current PCIE subtree. It should be noted that there are multiple ports in the PCIE switch, each port has a P2P bridge, and there is a virtual bus inside, which is configured according to the data manual during configuration.

There are many PCIE devices in the PCIE subtree managed by the bridge. These PCIE devices may have their own memory address space that needs to be accessed by the CPU. By configuring the Mernoy Limit and Memory Base registers of the bridge, access to the target device space can be achieved. These two registers are used to store the base address and size of the memory address space set of all devices on the PCIE subtree.

3 Hardware Design

The hardware of the multi-FC daughter card test equipment consists of a CPU module, a PCIE switching circuit, a motherboard, and an adapter board. The above components are integrated in the chassis. The following mainly describes the CPU module and the PCIE switching circuit:

3.1 CPU module design

In this test equipment, the CPU module is based on the MPC858 processor and is implemented by connecting FLASH, memory, PCIE interface, network port, serial port and other interface control circuits. As shown in Figure 3, the embedded real-time operating system VxWorks is transplanted into the CPU module, the device driver is secondary developed, and then VxWroks is used as the software development platform to develop the test program and realize the test task of the board.

3.2 PCIE switching circuit

The PCIE switching chip uses PLX's new generation of non-blocking, low-latency switching chips, which supports 48Lane and 12 PCIE ports. Through flexible hardware configuration and software programming, PCIE switching supports multiple port configurations. In this design, the switching chip is configured as a transparent bridge, and because the FC daughter card is a 4Lane PCIE interface, it is configured with 1 4Lane upstream port and 11 4Lane downstream ports, which can meet the functional requirements of the test equipment.

4 Software Design

According to the functional requirements and hardware platform of the FC daughter card test equipment, the test software runs on the VxWorks5.5 operating system, which mainly includes three parts of software: PCIE configuration software, FC driver software and FC test software. The PCIE configuration software is mainly used to configure the CPU, PCIE switching chip and PCIE interface of the FC daughter card. Only through configuration can the FC daughter card be accessed from the CPU side. As the driver software of the FC daughter card, the FC driver software must access the hardware resources of the FC daughter card, control the communication of the FC daughter card, and manage functions through the PCIE interface after the PCIE configuration is successful. The FC test software completes various functions and performance tests of the FC daughter card by calling the interface provided by the driver software. Its hierarchical structure is shown in Figure 5.

4.1 PCIE Configuration Software

The main function of the PCIE configuration software is to enumerate the devices in the PCIE bus tree. Through PCI compatible configuration, each FC daughter card device can be accessed from the CPU side. In this design, only the PCI compatible configuration mechanism is used. The configuration process is similar to that of PCI devices. It mainly needs to configure the MPC8548 and PCIE switch, including the access to the configuration space, how to discover devices, and access to the device space.

In this design, the bus example of the test equipment is as follows:

Finally, the BAR space of the FC daughter card is mapped according to the bus where the FC daughter card is located and the allocated memory address space, so that the FC device can be accessed.

4.2 FC driver software

The FC driver software runs on the host and mainly provides interfaces for access control of FC devices, FC communication, clock synchronization, network management and other functions, which are used for upper-layer software calls to meet specific application requirements. In this design, after the test software completes the PCIE configuration, it accesses the FC daughter card by calling the driver software interface, completes the daughter card initialization, and then performs related tests.

4.3 FC test software

FC test software mainly uses the platform built by the test equipment to perform various tests on FC daughter cards. It can realize the function and performance test of a single FC daughter card, and can also realize the communication test of multiple FC daughter cards in the test environment. By calling the driver software, the test software can complete the following tests: including FC daughter card hardware resource test, FC protocol compliance test, communication function test, clock synchronization function test, network management function test, FC communication bandwidth test and other function tests.

5. Verification

In order to verify whether the test equipment meets the design requirements, it is necessary to test the communication functions of multiple FC daughter cards. In the verification environment, run the test software to perform FC-AE-ASM communication test. FC data is generated from the CPU side, DMAed to the inside of the FC daughter card through the PCIE interface, and then sent out from the FC daughter card, passed through the analyzer, received from the receiving channel, and then DMAed from the FC daughter card to the CPU side to compare whether the sent and received data are consistent. The test environment is shown in Figure 7. Through the serial port printing and the data captured by the analyzer, the FC daughter card runs stably and reliably, verifying that the test equipment fully meets the requirements of laboratory test verification, and the test equipment has been used for experimental testing of FC daughter cards, and the test results also meet the test requirements.

6 Conclusion

This article introduces a multi-FC daughter card test device based on PCIE interface, explains the overall design scheme, hardware composition and software design, and verifies the good feasibility and stability of the device through testing. The device has a significant effect on improving equipment test efficiency and reducing scientific research and production costs, and has good versatility and scalability. It has a good reference and reference role for other types of PCIE/PCI interface devices.