Application of new intelligent network monitoring system on intelligent container trucks

In accordance with the "Shanghai Intelligent Connected Vehicle Road Test Management Measures" jointly formulated by the Shanghai Economic and Information Technology Commission, the Shanghai Public Security Bureau and the Shanghai Transportation Commission, the entity that proposes and organizes the application for intelligent connected vehicle road testing or demonstration application must have the function of online monitoring of vehicle status and be able to transmit the following data information items 1, 2 and 3 in real time.

It can automatically record and store the following data information at least 90 seconds before a vehicle accident or failure occurs:

1. Vehicle control mode; 2. Vehicle position; 3. Vehicle speed, acceleration, etc.; 4. Environmental perception and response status; 5. Real-time status of vehicle lights and signals; 6. 360-degree video monitoring of the vehicle's exterior; 7. In-vehicle video and voice monitoring of the driver and human-machine interaction status; & Remote control commands received by the vehicle; 9. Vehicle failure conditions.

Traditional monitoring equipment is mainly based on video, and is basically a monitoring equipment that uses both analog and digital as well as analog. At present, under the leadership of the development of smart cars, vehicle-mounted monitoring equipment is moving towards diversification, intelligence, and digitalization. Conventional vehicle-mounted monitoring systems are widely used in my country, gradually transitioning from the traditional single-end mode to centralized and unified cloud-based monitoring.

According to Xi'an Public Transport Co., Ltd., the company uses on-board monitoring equipment to achieve comprehensive monitoring of vehicles, allowing relevant managers to promptly grasp the situation inside and outside the vehicle, ensuring the safety of pedestrians and the unified management of public transportation. Depending on the type of bus, 4_6 camera monitoring videos, vehicle CAN bus data, GPS positioning data records and cloud monitoring functions are used.

Due to the long body, large blind spot, bumpy operation, and detachable rear trailer of container trucks, the existing equipment needs to be applied to container trucks to achieve full-time multi-camera and 360-degree monitoring of the body without blind spots, which is somewhat difficult. The requirements for the intelligent network monitoring system applied to intelligent container trucks are as follows:

(1) It is necessary to conduct all-round video surveillance inside and outside the container truck

(2) It is necessary to access multiple camera signals for simultaneous acquisition;

(3) The data storage module needs to consider the risk of damage caused by vehicle bumps;

(4) The detachable trailer requires the monitoring equipment to adaptively process the change in the number of cameras;

(5) While providing multiple sets of video signals, it is necessary to adapt to the development of smart cars, which is reflected in the need to collect multiple CAN data and real-time GPS data, and achieve synchronization between different data;

(6) At the same time, the corresponding data needs to be reported to the cloud platform to realize remote monitoring function.

The hardware part of the new intelligent network monitoring system is shown in Figure 1.

It consists of the main processing chip, camera access module, CAN processing module and various peripheral devices:

(1) Hardware design: 3 sets of 4-in-1 camera connection modules, which can support simultaneous access of up to 12 cameras;

(2) The system uses the master and slave MCUs for parallel processing to improve the processing speed. The two processors use SPI for data transmission and clock synchronization. The master MCU needs to record all monitoring data (audio, video, CAN) and communicate with the cloud platform. Therefore, the chip selection needs to consider the encoding capability and CPU processing capability. The current system uses NVIDIA TX2. The slave MCU needs to weigh the number of CANs, CPU main frequency, and vehicle level. The chips selected during the system design process include NXP's MPC57XX, MPC56xx, MPC55xx; Infineon's XMC 1400, TC29xT, TC37xTX and other chips.

(3) The CAN processor performance supports up to 6 CAN groups to access simultaneously, and has subsequent expansion capabilities;

(4) A backup rechargeable battery is designed for the power supply to ensure data recording after the vehicle loses power, meeting regulatory requirements;

(5) The system provides two storage device access interfaces. The data transmission protocol supports 4G, WIFI, and Gigabit Ethernet. The system also supports ublox/M8L GPS positioning functions with different precisions.

1.2 System Architecture

The system framework of the new intelligent network monitoring application is shown in Figure 2. The audio data, GPS data and multiple sets of camera data are processed by the data receiving module, and the original camera data is directly processed by the GPU. In this process, each camera data needs to be copied and converted once.

As shown in Figure 3, the memory space is maximized through timing control. The data is finally dynamically spliced ​​through the video memory to form a multi-camera image, and then transmitted to the HDMI display module and Recorder (data recording module). The HDMI communication interface can display the real-time camera image on the screen to facilitate the driver or relevant staff to observe the situation inside and outside the vehicle warehouse.

Recorder uses a timer to synchronize frames of all input data. The timer sets the frame rate according to the current system settings. It intercepts and splices the camera data and audio of each frame and performs H264 and audio encoding to generate a video file. The encoded data is pushed through the RTMP protocol (Real Time Messaging Protocol) to push audio and video data. The RTMP protocol is used to transmit audio, video and data between platforms, and its implementation is based on the TCP-IP protocol.

Therefore, it can provide reliable interaction and no packet loss occurs during transmission on the Internet, thus ensuring that the cloud platform and monitoring APP can perform real-time, reliable, and low-latency streaming display through the network. The raw CAN data and GPS data obtained between frames are compressed by the protocol to record the current frame number and timestamp and other data and store them into protocol files.

Data post-processing mainly provides cloud interaction, equipment maintenance, and scene annotation functions. The APP-side monitoring program can play video data in real time through WIFI connection. When an abnormal situation occurs during vehicle operation, the driver can use the APP's point-marking function to upload the 30 seconds audio and video and protocol data before and after the point-marking moment to the platform server. The administrator can quickly locate and analyze the uploaded fixed scene data.

The cloud interaction function provides handshake authentication between the device and the cloud platform, and issues platform commands (such as restart, OTA upgrade, etc.). At the same time, it collects the current main performance status and abnormal status of the system and reports them to the platform to realize daily maintenance functions.

Monitor is a set of monitoring modules to ensure the normal operation of the equipment. Its function is to check the current application running status and report abnormalities to the platform in time, so that the monitoring personnel can grasp the vehicle status in time. At the same time, this module has certain emergency response mechanisms, such as restoring application operation, repairing hard disk damage that may be caused by bumps, and switching redundant storage devices, etc., to maximize the integrity of monitoring data recording.

1.3 Monitoring Performance

The new intelligent network monitoring system provides a video recording bit rate of up to 3000Mbps. The hardware is designed with 12 LVDS camera interfaces, of which 4 are used for subsequent upgrades and backup. The current software performance can support 8 camera accesses at the same time, providing high-definition video materials for monitoring. The platform streaming bit rate can reach 100Mbps, 20fps. At the same time, CAN reception can support 6 groups of CAN access at the same time, and can simultaneously record multiple groups of sensor signals and multiple groups of vehicle CAN signals, which is convenient for subsequent data analysis and monitoring. The device supports access to a 1Tb hard disk as a primary storage backup, and an SD card as a redundant storage backup to ensure data recording integrity.

The cloud platform can obtain the online status of the equipment in real time, and locate the vehicle in real time when online. At the same time, the platform can remotely monitor in real time. The CAN data of the vehicle and the sensor can be uploaded to the platform through customer-defined signals to achieve overall monitoring. The device has an independent power-off storage module, which can ensure that the system can continue to work for more than 3 minutes after the vehicle is turned off, and ensure data recording after the vehicle accidentally loses power. It can automatically record and store data at least 90 seconds before the vehicle accident or failure occurs to meet regulatory requirements.

The application example of the new intelligent network monitoring system on the intelligent container truck is shown in Figure 4. The system is connected to 8 cameras (5 horizontal fovl92°, vertical fovl34°, 3 horizontal fovl20°, vertical fovl05°, all with a resolution of 1280x720), and 6 groups of cameras are arranged outside the cabin. The front of the vehicle and the rear trailer use horizontal 120.

Cameras (1, 6) are mainly designed to monitor areas far ahead and behind, so cameras with less distortion are used. The vehicle body uses four horizontal 192° cameras (2--4) to cover blind spots. A 120° camera (7) is installed on the A-pillar in front of the co-pilot seat of the container truck to monitor the driver and passengers in the cabin. A horizontal 192° camera is used at the connection between the front of the vehicle and the trailer to monitor the blind spots that occur when the front of the vehicle deviates from the body when the trailer turns.

Figure 5 shows an example of vehicle CAN data connection: OBD and power CAN can provide real-time vehicle status monitoring (such as steering wheel angle, vehicle speed, gear position, etc.); the intelligent driving CAN channel (hereinafter referred to as intelligent driving CAN) provides vehicle driving mode information (manual driving/automatic driving); RTK+IMU provides precise positioning information, and Radar provides obstacle location detection information. Both are important components of the perception system commonly used in intelligent driving, and are also important data sources for decision-making and control. This new intelligent network monitoring system can record all data through CAN bus signal monitoring (Listen only) and push it to the cloud platform according to customer-defined signals, improving the safety factor for intelligent travel.

Summarize

In view of the characteristics of container trucks, such as long body, large blind spot and detachable rear trailer, this paper proposes a vehicle-mounted monitoring system suitable for intelligent container trucks, which can realize multi-channel high-definition video outside and inside the cabin without blind spots, multi-channel CAN channel monitoring data recording for intelligent driving, and cloud monitoring. The system can currently provide 8-channel camera data (1280X720 resolution) and 6-channel CAN channel simultaneous acquisition.

At present, the system is widely used in intelligent container trucks. The cloud platform generates 2,000-3,000 kilometers of mileage monitoring data every day, and the scale is constantly expanding. According to statistics of monitoring equipment in normal operation, the total online time of all equipment is within 2,800 hours, and the system failure rate is less than 5%, of which 60% are accessory failures.

In the actual application process, there is a demand for expanding the number of cameras and CAN channels of the monitoring equipment. In addition to expanding the channels, the next optimization direction of the system is to access more communication protocol sensor data monitoring such as LIN, TCP-IP, etc.

(Click on the picture to get conference information)