Design and Application of Distributed System of CPCI Bus

Large-scale systems should adopt a distributed network architecture and be open and scalable to adapt to the ever-changing application environment and needs. Each module in the system should be responsible for processing different types of data and should be relatively independent and autonomous. At the same time, they should be interconnected at different levels to achieve mutual access and collaborative work. The system should also have good integration, requiring an effective component construction framework at the functional level and a unified data interaction platform at the component level [1].

Based on the above analysis, we choose CPCI bus as the data communication platform of the distributed system. CPCI bus technology is a combination of PCI bus technology and mature European card assembly technology. In terms of electrical, logical and software functions, it is fully compatible with PCI standards and breaks through the limitation of 4 slots of PCI standard. Compared with VME bus modules, it is low-priced and has the advantages of openness, easy expansion and high density. At the same time, it can achieve 99.999% high availability. The use of CPCI bus technology and hardware interface design specifications can utilize its multi-module plug-in card design advantages to support distributed processing of multiple services and realize seamless connection of modular data processing units, providing high-speed and reliable guarantees for distributed data exchange. It is very suitable as a distributed system business processing communication platform and is also suitable for wide application in communication and embedded systems [2-4].

This paper presents a communication system design based on CPCI bus. The system adopts a distributed network architecture and supports the processing and data interaction of multiple packet switching services. The paper first presents the system structure and principle design, and proposes a message storage and address information maintenance strategy based on the "drawer mechanism" to address the difficulties of cross-bus communication of distributed business processing modules. It also describes the implementation of key technologies such as data interference-free transmission, and finally gives a technical summary and outlook.

2. Overall system design

2.1 System structure characteristics

The distributed system structure we designed is shown in Figure 1. Different equipment boards in the system independently process the corresponding business data, convert it into unified IP data for intercommunication, and maintain their own routing tables to independently complete data forwarding. The system distributes the interfaces with specific business networks to various equipment boards for standard access channel adaptation, and distributes various types of network data to various boards for processing and forwarding, realizing the perfect combination of centralized configuration and distributed access and data processing.

In the CPCI distributed bus architecture, the backplane provides physical connection and circuit guarantee for bus switching. The system slots on the backplane provide bus arbitration, clock distribution, and restart of each board on the backplane. The peripheral slots can be used to place simple interface boards, intelligent slave devices, or bus control devices [2,3]. Each CPCI board has a processor and an embedded real-time system. The processor uses Motorola's PowerPC-860. At the same time, PLX's 9054 and 9056 PCI bridge chips are used to build an efficient and stable transmission bridge between PowerPC-860 and the CPCI bus. The PLX 9054/9056 chip realizes the function of the CPCI master control device, supports PCI2.2 protocol, simplifies the design of connecting to PowerPC, has good compatibility, and can be easily expanded to a 66MHz clock and 64-bit PCI bus. In particular, the PLX 9056 has an embedded bus arbitrator, which can reduce the system scale and make the system more stable [5].

Figure 1 Data communication system structure diagram

2.2 System resource sharing and information exchange

The system adopts a CPCI-based single-bus multi-processor/multi-operating system distributed architecture. Each board in the system has an independent CPU and operating system, address and memory space, and independent I/O and interrupts. It can independently complete data operations, and each board can be regarded as a computer host. The topological structure formed by the distributed system is a fully connected network, in which each node can directly access other nodes; from the perspective of CPCI bus transmission, the boards on all slots are equal and can act as masters to actively initiate bus transmission. For this bus-based distributed architecture, we designed a cross-bus memory access mechanism to map the system memory or device memory (such as memory expansion cards) of other boards in the system to the local address space, and then access the mapped memory in the same way as the system memory, so that each board can access the memory resources of other boards on the bus.

2.3 Unified and standardized access interface

Heterogeneous networks are connected to distributed systems through standard channel adaptation. Non-IP data such as voice, X.25, and serial port data are converted into IP data through data adaptation modules. The embedded real-time systems of each board in the system process and interact with the data. Various heterogeneous networks are connected to the corresponding network devices in the embedded system. The network device driver calls the unified interface provided by the CPCI bus driver to realize data transmission between the real-time system and the bus. When sending data, the network device driver controls the bridge chip through the bus driver to perform address conversion, data forwarding, interrupt generation, etc., and generates corresponding bus operations to send data to the bus; when receiving data, the bus driver responds to the interrupt, receives the data of the corresponding address segment on the bus, and performs data parsing, address conversion, data forwarding, and other interrupt generation operations in the interrupt service program. We use the Linux operating system, whose network system is mainly based on the UNIX socket mechanism. Data is transferred between the system protocol stack and the driver through a special data structure (sk_buff). The data transmission process between the real-time system kernel and the CPCI bus is shown in Figure 2:

Figure 2 Data transmission flow chart

3. Key technologies

3.1 “Drawer Mechanism” for Message Storage

Each board in the system shares a CPCI bus. We propose a message storage strategy based on the "drawer mechanism" to ensure interference-free data transmission between boards. During the initialization phase of adding boards to the system, the system board allocates an independent PCI bus address interval for each board on the bus. When other boards send data to it, they write the data to the specified address area. A board will receive data from different boards. To avoid interference caused by each board sending data to the same base address, the same board address area allocates independent read and write spaces of the same size to other boards. We name it "drawer". In this way, data from a board will be sent to its corresponding "drawer". Each time the data is stored in sequence instead of overwriting to ensure the data processing time of the board. When the data length exceeds the remaining space in the drawer, it is stored from the beginning like a ring buffer. The "drawer mechanism" is shown in Figure 3. The left square represents different boards on the bus, and the right is the PCI bus address space. The corresponding address range of board B is from point a to point e, where the space between points a and b is only used for data transmission from board A to B, the space between points b and c is only used for data transmission from board C to B, and so on.

Figure 3 Board data receiving "drawer"

Based on this message storage mechanism, we define several address tables to maintain address information related to data transmission. A static base address table is maintained on the system board, which records the pre-assigned base address of the board in each slot. A board address mapping table and a transmission address offset table are maintained on all boards. The board address mapping table is a structure array, each item in the array represents a card slot, which contains address information such as board name, slot number, base address and address range for configuration during data transmission. Its data structure is as follows:

typedef struct _BUS_ADDR_MAPPING_INFO{

char board_name[BOARD_NAME_LENGTH];

int slot_number;

unsigned long base_addr;

unsigned long range;

｝BUS_AddrMapping_Info, ＊P_BUS_AddrMapping_Info;

The transmission address offset table is an unsigned integer array used to record the address offset of each board during data transmission between boards. The initial value is zero. After each transmission, the address offset of the receiving board increases by the data transmission length. When the address space is not enough to store the data to be transmitted, the offset address is set to zero and rewritten from the beginning of the area. Its data structure is defined as follows:

u32 current_offset_table[NUM_OF_SLOT] = {0, 0, 0, 0, 0, 0, 0, 0};

3.2 Data transmission implementation

We define a data structure IPH (Internal Packet Header), which contains attributes such as data type, length, source card slot number, etc. It is used as a packet header to encapsulate the message before transmitting data, so that the receiver can distinguish and process according to the data service type after parsing the packet header. The main IPH types include board configuration information, port registration information, routing information, unknown data types, etc. The data structure iph_attr is defined to distinguish different IPH_info types, located at the beginning of the data packet, and its data structure is as follows:

typedef struct _IPH_ATTR {

u32 board_id; /*from which board*/

int iph_type; /*datagram type*/

unsigned long length; /*datagram length (without IPH)*/

}IPH_ATTR, *P_IPH_ATTR;

Different data structures are defined for various types of IPH information and are stored in the data packet header after the iph_attr structure.

When sending data, the data is encapsulated in IPH, the destination PCI address is selected according to the aforementioned board address mapping table, and then the bus interface function is called to complete the data transmission. The sender notifies the receiver of the transmission address and data length information by writing to the mailbox register of the receiving board bridge chip, generating an interrupt to trigger reception. The PLX bridge chip supports direct access of the local bus to the PCI bus. It has 8 mailbox registers, the first four of which can generate interrupts. Each mailbox is 32 bits. The transmission address and data length information are respectively used in mailbox i and mailbox i+4. In this way, when the receiver receives two parameters, an interrupt is generated to receive data [5]. This mechanism enables the receiving process to have four service windows, which improves the system throughput.

When the parameters are written to the mailbox of the PLX chip of the receiving board, a local interrupt is generated to check the "drawer". Before the interrupt is generated, the data has actually been sent to the target board. The interrupt service program maintains a data queue for the receiving end. It reads the information in the mailbox, analyzes the address to find the corresponding data and sends it to the bottom half for processing. The bottom half parses the IPH of the data packet to distinguish the data type. If it is configuration, port, routing and other information, it will be configured accordingly. If it is data information, it will be processed or forwarded.

In summary, the system completes the mapping of PCI address space between boards through the "drawer mechanism" and the maintenance of several address tables. When the board writes data to the mapped address space, it can transfer the data to the target board through the bus, thus realizing cross-bus memory access of the board; the customized IPH data packet header is used to distinguish data types to assist in data information management, completes routing maintenance, and the logical functions of the forwarding engine, thus realizing interference-free data transmission and effective communication management.

4. Summary and Outlook

The innovation of the author of this article is: a CPCI-based distributed system design is given, and a message storage mechanism and address information maintenance strategy based on the "drawer mechanism" are proposed. The CPCI-based distributed system described in the article can achieve a 64-bit bus width and a peak bandwidth of 264MB/s. Each host in the system can independently complete data processing and communication, and can carry multiple voice and data services. Users can also communicate data through the PSTN network connected to the user access board and the voice service board and the Internet connected to the data service board. It has great application prospects in the fields of communications and military. In order to make this communication system more practical on a large scale, future work includes:

(1) Implement an easy-to-operate remote management system to monitor and coordinate communication services;

(2) Design CPCI interface cards that support more service types, such as xDSL, H.264, etc.;

(3) In the case of significant external interference, strict performance testing is carried out to prove that the system can meet telecommunications-grade business requirements.