[AI Challenge Camp 3] Deploy MNIST digit recognition on Luckfox Pico

北方

[AI Challenge Camp 3] Deploy MNIST digit recognition on Luckfox Pico [Copy link]

Deploy MNIST digit recognition on Luckfox Pico

1 Overview

After completing the first and second tasks, you are ready to deploy the AI model on Luckfox.

#AI Challenge Camp First Stop#Pytorch Installation and MNIST Recognition - Linux and Android - Electronic Engineering World - Forum (eeworld.com.cn)

[AI Challenge Camp Second Stop] Convert MNIST to RKNN format and package the algorithm into SDK for engineering deployment - Linux and Android - Electronic Engineering World - Forum (eeworld.com.cn)

2 Update firmware

When I first got the development board, I could connect it with adb and read the file system, but after connecting the CSI camera, I could not read the video as described in the manual. According to the following wiki information,

CSI Camera | LUCKFOX WIKI

The factory version is a test image, which can only test the normal operation of the development board. To implement any specific function, you need to compile the system yourself. There are buildroot and ubuntu options. Ubuntu is larger and needs to be loaded on the SD card, while buildroot is much simpler and can be used with menuconfig definitions and then compiled by yourself. Therefore, choose to use buildroot to implement it and write it directly to the SPI memory.

According to the manual, you need to clone the source code from GitHub, configure the cross-compilation tool, and use the script to compile. Because gitee also synchronizes the source code, you can quickly get it done from gitee, avoiding the long wait on github.

Because the cross-compilation tool is only applicable to Linux systems, it is more convenient to do this step directly on Ubuntu, otherwise WSL or a virtual machine is required.

The following is the compilation process. On an old laptop, it took about 4 hours, and there were only some warnings during the whole process. It was very smooth.

The compilation result is as shown below, and the build output path is prompted.

Enter the directory and generate the following files. There is only one file that needs to be written, which is uboot.img

In fact, the cloud disk also provides an image that can be written directly. You can download and unzip it. See below, which is actually the same as the above

Press the Boot button to power on, and you can find the development board, as shown in Fuzhou Rockchip....

Then use the upgrade_tool tool to download to the Luchfox development board. The upgrade_tool also needs to be downloaded and installed. It is only suitable for Linux systems.

update successed

At this time, I searched for the connected network port and found that I could not connect to the development board using 172.32.0.93.

This took a long time, and finally I found that this was a Luchfox pico pro, and the network port was configured to another URL starting with 192.168.10.x. You need to use RNDS to connect to the virtual network port of the USB port before you can continue to connect using this URL.

Although not practical, implementing the above functionality on Windows requires the use of another tool,

I won’t repeat myself, just check the wiki above.

3. Realize video stream capture

Connect the video camera. Note that the connection of Luchfox Pico Pro is different from that of ordinary Pico. Use VLC, which can directly capture the video stream.

Select the real-time video of channel 0 of rtsp://172.32.0.93, and the display is as follows: flowers and plants

You can also use ssh to connect, username root, password luckfox, and MobaXterm in GUI mode, which is much more intuitive.

To upload files, you can also use the scp command line command.

Similarly, adb can also be used. First search for the device. If there is only one, use adb shell. This adb is the android device connection tool. If you have installed android studio, it comes with it. Otherwise, you need to download it separately and unzip it for use.

Find rkipc.ini in this directory. This means that the CSI camera is successfully connected. The original factory test version does not have it.

Use adb push to upload files

This meets the requirements for video recognition MNIST.

4 Call SDK to implement MNIST digit recognition based on video stream

4.1 As mentioned earlier, you first need to train your own AI model, then convert it into the public ONNX format, and then use professional tools to compress it and convert it into the rknn format. After this compression, the parameters are all integerized, and there is a certain loss in recognition accuracy.

This code is written in C language. The video data is read using the opencv library. It is compiled using the lib library.

void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk);  
cv::Mat frame(height,width,CV_8UC3,data);   

The mat matrix data is read and the output is read under the inference command. This is the SDK compiled earlier.

inference_mnist_model(&rknn_app_ctx, sub_pics[i], detect_results);

In general, the above inference commands use the built-in AI data processing API to implement the corresponding high-intensity convolution calculations, output the results, and compare them with the results of the selected numbers.

This is based on the changes of the yolo8 example code. Other models are still implemented using the programming sequence developed in this way. The only difference is the model parameters used.

In this way, through the calculation of different models, the code changes are almost the same, but by separating the trained models, multiple functions can be achieved.

4.2 After compiling, the executable file is used directly, and the result file is interrupted.

This is the automatic call camera that needs to be stopped by RkLunch-stop.sh. Note the lowercase and uppercase letters.

Restart the code and the output command line of continuous recognition will be displayed.

As you can see from the output, the substandard 2 is recognized as 3

The substandard 3 is identified as 5.

This time the recognition is correct, the handwriting is relatively standard 3, and it doesn't matter if the font is a little blurry. And the recognition speed is very fast, it can reach 21fps, which is 30fps for continuous video. Basically, if you don't pay too much attention, you can recognize data in real time.

5 Summary

After this process, we can find that MNIST can realize the digital recognition function in a relatively practical way, and the speed is also quite good. It should be noted that the MNIST model generally needs to be compressed. The recognition rate is obviously not as good as when testing. The general confidence level is 50%, while it has exceeded 90% during training. In the first stage of training, it is actually 99%. The data looks very poor, but the more standardized writing can be perfectly read.

This is a practical choice that balances cost performance and power consumption.

Jacktang

Are there hardware limitations due to the size and complexity of the model?

北方

Jacktang posted on 2024-6-14 07:20 Will there be hardware limitations due to the size and complexity of the model?

It is indeed the case.

Usually, the smallest industrial model is around 4-5M, such as Google's mobileNet v2. This requires a large amount of data throughput and computation, so compression is required.

Usually, the floating-point number of the parameter is changed to an integer, which can compress it by 2-4 times. Other structural optimizations require more powerful optimization capabilities.

This MNIST effect is good, reaching 20fps, the accuracy is much lower, around 40%-50%, but it can still be recognized.

It's very successful.