ADAS target recognition based on Vitis AI

1. Basic knowledge

1. Viti s ™ AI Development Environment

The Vitis™ AI development environment accelerates AI inference on Xilinx hardware platforms, including edge devices and Alveo™ accelerator cards. This environment consists of optimized IP cores, tools, libraries, models, and design examples. It is designed with efficiency and ease of use at its core to fully explore the full potential of AI acceleration through Xilinx SoCs and Adaptive Compute Acceleration Platforms ( ACAP ). The Vitis AI development environment abstracts the complex details of the underlying programmable logic, helping users without FPGA knowledge to easily develop deep learning inference applications.

2. Vitis AI Model Zoo

Vitis AI has a very important tool: Vitis AI Model Zoo, which is similar to Vitis AI's model mall .

Vitis AI Model Zoo contains optimized deep learning models that accelerate the deployment of deep learning inference on Xilinx platforms. These models cover different applications, including AD AS/AD, video surveillance , robotics , and data centers . Users can start with these pre-trained models and enjoy the many benefits of deep learning acceleration.

3. ADAS

The Advanced Driving Assistance System (ADAS) uses various sensors ( millimeter wave radar , laser radar, single and binocular cameras, and satellite navigation) installed on the car to sense the surrounding environment at any time during the driving process, collect data, identify, detect and track static and dynamic objects, and combine navigation map data to perform systematic calculations and analysis, so as to allow drivers to be aware of possible dangers in advance, effectively increasing the comfort and safety of car driving. In recent years, the ADAS market has grown rapidly. Originally, this type of system was limited to the high-end market, but now it is entering the mid-range market. At the same time, many low-tech applications are more common in the entry-level passenger car field. Improved new sensor technologies are also creating new opportunities and strategies for system deployment.

2. Environment Construction

There are two ways to install the Vitis AI library. One is to reconstruct the system by configuring PetaLinux, and the other is to install the Vitis AI library online. After installing the Vitis-AI library, install the Vitis-AI dependent library.

1. System Download & Installation

If you haven't played with PetaLinux before, then give it a try. First download the PetaLinux system image from the official website (https://china.xilinx.com/member/forms/download/design-license-xef.html?filename=xilinx-kv260-dpu-v2022.2-v3.0.0.img.gz ). Please note that you must register an AMD account first, and then fill in some information to successfully register. The compressed file is 3.3G, and the decompressed file is 8.8G.

Pay attention to the file naming. The Vitis AI version is V3.0.0. There is a pitfall here. I will tell you later if you encounter it. After burning the IMG file to the TF card and powering on, PetaLinux is like this:

root@xilinx-kv260-starterkit-20222:~/Vitis-AI# uname -a

Linux xilinx-kv260-starterkit-20222 5.15.36-xilinx-v2022.2 #1 SMP Mon Oct 3 07:50:07 UTC 2022 aarch64 aarch64 aarch64 GNU/

root@xilinx-kv260-starterkit-20222:~/Vitis-AI#

What comes into view are two folders, including the famous Vitis-AI:

Next, we will have a lot of fun playing with this folder.

(II) Preparing image packages

Download vitis_ai_runtime_r3.0.0_image_video.tar.gz from the official link (https://china.xilinx.com/bin/publ ic /openDownload?filename=vitis_ai_runtime_r3.0.0_image_video.tar.gz), which includes the image and video files required by the demo. After downloading, decompress it and set it aside.

3. ADAS Target Recognition

Vitis AI provides many examples, including a demo for ADAS target recognition. In Vitis-AI/examples/vai_runtime/adas_detection, the executable CPP program has been compiled and can be executed directly.

Before executing the demo, read the readme first.

Before running the program, please download the corresponding model and install it.

The model required by this sample is: yolov3_adas_pruned_0_9

You can find the detailed informantion of this model under

Vitis-AI/models/AI-Model-Zoo/model-list/dk_yolov3_cityscapes_256_512_0.9_5.46G_1.3/model.yaml

In the model.yaml, you will find the model's download links for different platforms.

Please choose the corresponding model and download it.

Take ZCU102/ZCU104 as an example, execute the following commands to download and install the model.

wget https://www.xilinx.com/bin/public/openDownload?filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r2.5.0.tar.gz -O yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r2.5.0.tar.gz

mkdir -p /usr/share/vitis_ai_library/models

tar -xzvf yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r2.5.0.tar.gz

cp yolov3_adas_pruned_0_9 /usr/share/vitis_ai_library/models -r

The non-KV260 content has been cut off. The readme basically tells users to go to Vitis AI Model Zoo to download the corresponding model and install it.

Then follow the guideline, copy the adas.webm file from the previous graphics package to the current directory, and then execute

./adas_detection adas.webm /usr/share/vitis_ai_library/models/yolov3_adas_pruned_0_9/yolov3_adas_pruned_0_9.xmodel

Hmm, something went wrong:

**Attention, there is a pitfall! **System prompt

Please re-compile xmodel

Is that right? Should I recompile the xmodel? After a closer look, I found that the system was running Vitis AI V3.0, but the xmodel was V2.5, which caused the CHECK fingerprint to fail. I just needed to download a V3.0 xmodel from Vitis AI Model Zoo. I did it right away!

root@xilinx-kv260-starterkit-20222:~/Vitis-AI/examples/vai_runtime/adas_detection# wget https://www.xilinx.com/bin/public/openDownload?filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz -O yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz

--2023-09-27 06:52:41--  https://www.xilinx.com/bin/public/openDownload?filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz

Resolving www.xilinx.com... 223.119.248.58, 223.119.248.90

Connecting to www.xilinx.com|223.119.248.58|:443... connected.

HTTP request sent, awaiting response... 302 Moved Temporarily

Location: https://xilinx-ax-dl.entitlenow.com/dl/ul/2023/01/06/R210771244/yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz?hash=PJrrJ06TWqMoH_m1gKVgXw&expires=1695855161&filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz [following]

--2023-09-27 06:52:41--  https://xilinx-ax-dl.entitlenow.com/dl/ul/2023/01/06/R210771244/yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz?hash=PJrrJ06TWqMoH_m1gKVgXw&expires=1695855161&filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz

Resolving xilinx-ax-dl.entitlenow.com... 223.119.244.25

Connecting to xilinx-ax-dl.entitlenow.com|223.119.244.25|:443... connected.

HTTP request sent, awaiting response... 302 Moved Temporarily

Location: https://amd-ax-dlf.entitlenow.com/dl/ul/2023/01/06/R210771244/yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz?hash=PJrrJ06TWqMoH_m1gKVgXw&expires=1695855161&filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz [following]

--2023-09-27 06:52:45--  https://amd-ax-dlf.entitlenow.com/dl/ul/2023/01/06/R210771244/yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz?hash=PJrrJ06TWqMoH_m1gKVgXw&expires=1695855161&filename=yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz

Resolving amd-ax-dlf.entitlenow.com... 223.119.248.34, 223.119.248.40, 2402:4f00:4002:400::df77:f828, ...

Connecting to amd-ax-dlf.entitlenow.com|223.119.248.34|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1875420 (1.8M) [application/octet-stream]

Saving to: 'yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz'

yolov3_adas_pruned_0_9-zcu102_ 100%[==================================================>]   1.79M  1.54MB/s    in 1.2s

2023-09-27 06:52:50 (1.54 MB/s) - 'yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz' saved [1875420/1875420]

root@xilinx-kv260-starterkit-20222:~/Vitis-AI/examples/vai_runtime/adas_detection#

Execute the following command again:

root@xilinx-kv260-starterkit-20222:~/Vitis-AI/examples/vai_runtime/adas_detection# tar -xzvf yolov3_adas_pruned_0_9-zcu102_zcu104_kv260-r3.0.0.tar.gz

yolov3_adas_pruned_0_9/

yolov3_adas_pruned_0_9/meta.json

yolov3_adas_pruned_0_9/yolov3_adas_pruned_0_9.xmodel

yolov3_adas_pruned_0_9/md5sum.txt

yolov3_adas_pruned_0_9/yolov3_adas_pruned_0_9.prototxt

yolov3_adas_pruned_0_9_acc/

yolov3_adas_pruned_0_9_acc/yolov3_adas_pruned_0_9_acc.prototxt

yolov3_adas_pruned_0_9_acc/yolov3_adas_pruned_0_9_acc.xmodel

root@xilinx-kv260-starterkit-20222:~/Vitis-AI/examples/vai_runtime/adas_detection# cp yolov3_adas_pruned_0_9 /usr/share/vitis_ai_library/models -r

4. ADAS Target Recognition Experience

1. Target Identification

Connect an HDMI display, keyboard, and mouse, and execute on KV260 (if executed on SSH or serial port, a cv::Exception will be prompted):

`./adas_detection video/adas.webm /usr/share/vitis_ai_library/models/yolov3_adas_pruned_0_9/yolov3_adas_pruned_0_9.xmodel`

At the end of the video, we can see that the recognition accuracy and real-time performance are good, and the FPS is maintained at around 40.

2. Dashboard monitoring

KV260 provides a Hardware Platform Statistics page on PetaLinux, which is quite interesting and is used to display the real-time consumption of system hardware resources. The monitored contents include CPU consumption, memory idle and consumption, voltage, temperature...