#AIChallengeCampTerminal#RV1106HandwrittenDigitRecognitionDeployment

天空中

#AIChallengeCampTerminal#RV1106HandwrittenDigitRecognitionDeployment [Copy link]

<i class="pstatus">This post was last edited by Sky on 2024-5-25 18:09</i> # RV1106 Digital Recognition## Briefly describe the use of RKMPI library to realize camera image capture, preprocessing, hardware encoding, combined with opencv-mobile for image processing, rtsp push streaming, and use VLC software to pull and observe images. ### Implementation effect* RKNN digital recognition and labeling
<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_811431_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>屏幕截图 2024-05-25 160227.png</strong> <em class="xg1">(427.36 KB, downloads: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >download attach</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_811431" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=811431&handlekey=savephoto_811431">save to album</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>
 ## Operation process RKMPI obtains image -> Use opencv-mobile to capture -> Opencv-mobile performs image preprocessing and selects numbers -> RKNN reasoning -> Results are returned## Brief description of the function library involved### opencv-mobile In embedded systems, many modules in opencv cannot be used, such as: opencv_gapi, opencv_videoio and other modules, so it is necessary to streamline openncv. opencv-mobile, a streamlined OpenCV library with a volume of only 1/10 of the official one, has been adapted to rv1106 and supports luckfox-pico MIPI CSI camera and rk-aiq/rga hardware acceleration [github library address](https://github.com/nihui/opencv-mobile) ### RKNN model conversion and deployment The RKNN-Toolkit2 tool provides C or Python interfaces on the PC platform to simplify the deployment and operation of the model. Users can easily complete the following functions through this tool: model conversion, quantization, reasoning, performance and memory evaluation, quantization accuracy analysis, and model encryption. The environment and installation of RKNN-Toolkit2 have been described in detail in previous articles. This article will not repeat them. ## Code explanation Thanks to [knv](https://home.eeworld.com.cn/space-uid-1362174.html) for sharing code and providing ideas and optimization methods. This code refers to the article [RKMPI Example Usage Guide | LUCKFOX WIKI](https://wiki.luckfox.com/zh/Luckfox-Pico/RKMPI-example#6-opencv-mobile-Marking Push Stream Frame Number Example) ### Convert the image obtained by RKMPI to opencv Mat type```c++ void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk); //Get the virtual address of the memory buffer block corresponding to the MPI data frame cv::Mat frame(height,width,CV_8UC3, data);//Convert to opencv Mat type to prepare for subsequent image preprocessing``` ### Image preprocessing```c++ cv::Mat frame64; cv::cvtColor(frame, frame64, cv::COLOR_BGR2GRAY); //Convert color image to grayscale image //Graphic binarization. Instead of using a fixed threshold, Otsu's method is used for dynamic threshold processing to increase the stability of digital recognition under different light sources //Use THRESH_BINARY_INV mode to set pixels greater than the threshold to 0 and pixels less than the threshold to 255 so that there is no need to perform pixel inversion cv::threshold(frame64, frame64, 100, 255, cv::THRESH_BINARY_INV | cv::THRESH_OTSU); ``` #### Find the outline of the number and select it```c++ // Find the outline and declare variables to store the outline std::vector<std::vector>>contours; std::vector<cv::vec4i>hierarchy; cv::findContours(frame64, contours, hierarchy, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_NONE, cv::Point()); //Declare variables to store the selected rectangle cv::Rect rect; int out_num = 10; for (int i = 0; i < contours.size(); i++) { rect = cv::boundingRect(contours<i>); //Fit a rectangle in the contour being searched for //Filter out the wrong rectangle by the length, width and aspect ratio of the rectangle if (rect.width < 100 && rect.width > 10 && (rect.height/rect.width)>1 && rect.height <200) { //Increase the size of the rectangle by 10 pixels rect.x = std::max(0, rect.x - 10); rect.y = std::max(0, rect.y - 10); rect.width = std::min(720 - rect.x, rect.width + 20); rect.height = std::min(480 - rect.y, rect.height + 20); //Draw a rectangle on the original image cv::rectangle(frame, rect, cv::Scalar(0,0,255), 2); //Extract the pixels in the area where the rectangle is located cv::Mat roi = frame64(rect); //Resize the image to 28x28 cv::resize(roi, roi, cv::Size(28,28), 0, 0, cv::INTER_LINEAR); //Copy the virtual address of the image to the tensor memcpy(rknn_app_ctx.input_mems[0]->virt_addr, roi.data, 28*28*1); //Run model inference_NUM_model(&rknn_app_ctx, out_num); } } ``` This code does not filter the image, because after testing, the filtering operation does not affect the recognition rate of the number under stable light conditions, and the filtering operation will traverse the entire image to reduce the frame rate, so the filtering operation is discarded. ## Model initialization and model inference use RKNN zero copy API Note: rv1106/rv1103 only supports zero copy API RKNN zero copy API is a set of functions provided by Rockchip neural network processing library (RKNN, Rockchip Neural Network), which aims to optimize data transmission efficiency and reduce memory copy operations, thereby improving the performance of deep learning applications based on Rockchip platform. In traditional memory operations, when data is transferred between different memory areas or different processes, data copying is often required, which consumes time and system resources. Zero copy technology tries to avoid these unnecessary copy operations. ### Model initialization```c++ //rknn context structure variable typedef struct { // RKNN running context, obtained after initialization by rknn_init function, is the core handle for performing model inference. rknn_context rknn_ctx; // Pointer to the maximum memory block, which can be dynamically adjusted as needed to accommodate inputs or intermediate results of different sizes. rknn_tensor_mem* max_mem; // Network memory pointer, usually used to store temporary data or intermediate results during model operation. rknn_tensor_mem* net_mem; // Input and output quantity structure, recording the number of input and output tensors of the model. rknn_input_output_num io_num; // Pointer to the input tensor attribute array, storing detailed attribute information of each input tensor (such as dimension, data type, etc.). rknn_tensor_attr* input_attrs; // Pointer to the output tensor attribute array, storing detailed attribute information of each output tensor. rknn_tensor_attr* output_attrs; // Input memory block pointer array, containing at least one element, used to store the input data of model inference. rknn_tensor_mem* input_mems[1]; // Output memory block pointer array, assuming that the model has 3 outputs, used to receive the result data of model inference. rknn_tensor_mem* output_mems[3]; // Number of model input channels. For image processing tasks, this usually refers to the number of color channels (such as 3 channels for RGB images). int model_channel; // Model input width, usually in pixels. int model_width; // Model input height, usually in pixels. int model_height; // Indicates whether the model is quantized. Quantized models are usually used to speed up inference and reduce memory usage. bool is_quant; } rknn_app_context_t; ``` ```c++ rknn_app_context_t rknn_app_ctx; // rknn structure variable int ret; const char *model_path = "./model/111.rknn"; //Model address memset(&rknn_app_ctx, 0, sizeof(rknn_app_context_t)); //Initialize the structure init_retinaface_model(model_path, &rknn_app_ctx); //Model initialization function printf("init rknn model success!\n"); ``` ```c++ /** * Function to initialize the RKNN model* * @param model_path path to the model file* @param app_ctx pointer to the application context structure, used to store model context and input and output related information* @return 0 if successful and -1 if failed */ int init_retinaface_model(const char *model_path, rknn_app_context_t *app_ctx) { int ret; rknn_context ctx = 0; // RKNN model running context // Use rknn_init function to initialize the model and pass in the model path ret = rknn_init(&ctx, (char *)model_path, 0, 0,NULL); if (ret < 0) { printf("rknn_init fail! ret=%d\n", ret); return -1; } // 查询模型输入输出次数 rknn_input_output_num io_num; ret = rknn_query(ctx, RKNN_QUERY_IN_OUT_NUM, &io_num, sizeof(io_num)); if (ret != RKNN_SUCC) { printf("rknn_query fail! ret=%d\n", ret); return -1; } printf("model input num: %d, output num: %d\n", io_num.n_input, io_num.n_output); // 获取和打印输入和输出的文化的详细信息rknn_tensor_attr input_attrs[io_num.n_input]; memset(input_attrs, 0, sizeof(input_attrs)); for (int i = 0; i < io_num.n_input; i++) { input_attrs<i>.index = i; ret = rknn_query(ctx, RKNN_QUERY_NATIVE_INPUT_ATTR, &(input_attrs<i>), sizeof(rknn_tensor_attr)); if (ret != RKNN_SUCC) return -1; dump_tensor_attr(&(input_attrs <i>)); // Print input tensor attributes} rknn_tensor_attr output_attrs[io_num.n_output]; memset(output_attrs, 0, sizeof(output_attrs)); for (int i = 0; i < io_num.n_output; i++) { output_attrs<i>.index = i; ret = rknn_query(ctx, RKNN_QUERY_NATIVE_NH WC_OUTPUT_ATTR, &(output_attrs<i>), sizeof(rknn_tensor_attr)); if (ret != RKNN_SUCC) return -1; dump_tensor_attr(&(output_attrs<i>)); // 打印输出网球属性} // Set input tensor attributes and allocate memory for inputinput_attrs[0].type = RKNN_TENSOR_UINT8; input_attrs[0].fmt = RKNN_TENSOR_NHWC; printf("input_attrs[0].size_with_stride=%d\n", input_attrs[0].size_with_stride); app_ctx->input_mems[0] = rknn_create_mem(ctx, input_attrs[0].size_with_stride); ret = rknn_set_io_mem(ctx, app_ctx->input_mems[0], &input_attrs[0]); // Allocate memory for output tensor and set memory attributes for (uint32_t i = 0; i < io_num.n_output; ++i) { app_ctx->output_mems<i> = rknn_create_mem(ctx, output_attrs<i>.size_with_stride); printf("output mem [%d] = %d \n",i ,output_attrs<i>.size_with_stride); ret = rknn_set_io_mem(ctx, app_ctx->output_mems<i>, &output_attrs<i>); if (ret < 0) return -1; } // Save model context and other relevant information to app_ctx app_ctx->rknn_ctx = ctx; app_ctx->is_quant = (output_attrs[0].qnt_type == RKNN_TENSOR_QNT_AFFINE_ASYMMETRIC); app_ctx->io_num = io_num; // Allocate memory and copy input and output attributes to app_ctxapp_ctx->input_attrs = (rknn_tensor_attr *)malloc(io_num.n_input * sizeof(rknn_tensor_attr)); memcpy(app_ctx->input_attrs, input_attrs, io_num.n_input * sizeof(rknn_tensor_attr)); app_ctx->output_attrs = (rknn_tensor_attr *)malloc(io_num.n_output * sizeof(rknn_tensor_attr)); memcpy(app_ctx->output_attrs, output_attrs, io_num.n_output * sizeof(rknn_tensor_attr)); // Determine the size information of the model based on the input formatif (input_attrs[0].fmt == RKNN_TENSOR_NCHW) { printf("model is NCHW input fmt\n"); app_ctx->model_channel = input_attrs[0].dims[1]; app_ctx->model_height = input_attrs[0].dims[2]; app_ctx->model_width = input_attrs[0].dims[3]; } else { printf("model is NHWC input fmt\n"); app_ctx->model_height = input_attrs[0].dims[1]; app_ctx->model_width = input_attrs[0].dims[2]; app_ctx->model_channel = input_attrs[0].dims[3]; } printf("model input height=%d, width=%d, channel=%d\n", app_ctx->model_height, app_ctx->model_width, app_ctx->model_channel); return 0; // 成功登陆 } ``` ### 模型因法```c++ /** * Function that executes model inference and processes output results* * @param app_ctx Pointer to the RKNN application context, which contains all the resources and parameters required for the model to run* @param out_num Reference parameter, the most likely category number after inference will be saved here* @return Success returns 0, failure returns -1 */ int inference_NUM_model(rknn_app_context_t *app_ctx, int &out_num) { int ret = 0; float probability = 0; // Use rknn_run function to perform model inference ret = rknn_run(app_ctx->rknn_ctx, NULL); if (ret < 0) { printf("rknn run error %d\n", ret); return -1; } // Get the pointer of model output data int8_t* output_data = (int8_t*)(app_ctx->output_mems[0]->virt_addr); // Create an array to store the dequantized output value float dequantized_values[10]; // Dequantize the output data for (int i = 0; i < 10; ++i) { dequantized_values<i> = (output_data<i> - (float)(-4)) * 0.122913; } // Apply the softmax function to convert the output value to probability applySoftmax(dequantized_values, 10); for (int i = 0; i < 10; ++i) { printf("Class %d: Probability = %.5f\n", i, dequantized_values<i>); // If the probability of the current category is greater than 0.5, check whether it is greater than the maximum probability found if(dequantized_values<i> > 0.5) { // Update the maximum probability and the corresponding category number if(probability < dequantized_values<i>) { probability = dequantized_values<i>; out_num = i; } } } return ret; } ``` ### Convert output value to probability```c++ void applySoftmax(float* data, int size) { // Find the maximum value in the array for numerically stabilized softmax calculation float max_val = data[0]; for (int i = 1; i < size; ++i) { if (data<i> > max_val) { max_val = data<i>; } } // Calculate the value of exp(xi - max_val) for each element and accumulate it for normalization float sum_exp = 0; for (int i = 0; i < size; ++i) { data<i> = exp(data<i> - max_val); // Subtract the maximum value from each value and perform exponential operation to prevent underflow caused by exponential operation sum_exp += data<i>; // Accumulate all exponential values for subsequent normalization } // Normalization step, convert all exp values into probability distribution (the sum of all probabilities is 1) for (int i = 0; i < size; ++i) { data<i> /= sum_exp; // Divide each element by the sum to get the probability value } } ``` ## Code running effect [local video]68fd412d539aec90db007434d4d9208e[/local video]app_ctx->model_channel); return 0; // Successful return } ``` ### Model inference function```c++ /** * Function that executes model inference and processes output results* * @param app_ctx Pointer to the RKNN application context, which contains all the resources and parameters required for model operation* @param out_num Reference parameter, the most likely category number after inference will be saved here* @return Success returns 0, failure returns -1 */ int inference_NUM_model(rknn_app_context_t *app_ctx, int &out_num) { int ret = 0; float probability = 0; // Use rknn_run function to perform model inference ret = rknn_run(app_ctx->rknn_ctx, NULL); if (ret < 0) { printf("rknn run error %d\n", ret); return -1; } // Get the pointer to model output data int8_t* output_data = (int8_t*)(app_ctx->output_mems[0]->virt_addr); // Create an array to store the dequantized output values float dequantized_values[10]; // Dequantize the output data for (int i = 0; i < 10; ++i) { dequantized_values<i> = (output_data<i> - (float)(-4)) * 0.122913; } // Apply the softmax function to convert the output value into a probability applySoftmax(dequantized_values, 10); for (int i = 0; i < 10; ++i) { printf("Class %d: Probability = %.5f\n", i, dequantized_values<i>); // If the probability of the current category is greater than 0.5, check whether it is greater than the maximum probability found if(dequantized_values<i> > 0.5) { // Update the maximum probability and the corresponding category numberif(probability < dequantized_values<i>) { probability = dequantized_values<i>; out_num = i; } } } return ret; } ``` ### Convert output value to probability```c++ void applySoftmax(float* data, int size) { // Find the maximum value in the array for numerical stabilization softmax calculationfloat max_val = data[0]; for (int i = 1; i < size; ++i) { if (data<i> > max_val) { max_val = data<i>; } } // Calculate the value of exp(xi - max_val) for each element and accumulate it for normalizationfloat sum_exp = 0; for (int i = 0; i < size; ++i) { data<i> = exp(data<i> - max_val); // Perform exponential operation on each value after subtracting the maximum value to prevent underflow caused by exponential operationsum_exp += data<i>; // Accumulate all exponential values for subsequent normalization} // Normalization step, convert all exp values into probability distribution (the sum of all probabilities is 1) for (int i = 0; i < size; ++i) { data<i> /= sum_exp; // Divide each element by the sum to get the probability value} } ``` ## Code running effect<div id="player_68fd412d539aec90db007434d4d9208e0" class="localvideo-box"></div><div class="localvideo-title">VLC media player 2024-05-25 16-54-12_Trim_Trim</div>app_ctx->model_channel); return 0; // Successful return } ``` ### Model inference function```c++ /** * Function that executes model inference and processes output results* * @param app_ctx Pointer to the RKNN application context, which contains all the resources and parameters required for model operation* @param out_num Reference parameter, the most likely category number after inference will be saved here* @return Success returns 0, failure returns -1 */ int inference_NUM_model(rknn_app_context_t *app_ctx, int &out_num) { int ret = 0; float probability = 0; // Use rknn_run function to perform model inference ret = rknn_run(app_ctx->rknn_ctx, NULL); if (ret < 0) { printf("rknn run error %d\n", ret); return -1; } // Get the pointer to model output data int8_t* output_data = (int8_t*)(app_ctx->output_mems[0]->virt_addr); // Create an array to store the dequantized output values float dequantized_values[10]; // Dequantize the output data for (int i = 0; i < 10; ++i) { dequantized_values<i> = (output_data<i> - (float)(-4)) * 0.122913; } // Apply the softmax function to convert the output value into a probability applySoftmax(dequantized_values, 10); for (int i = 0; i < 10; ++i) { printf("Class %d: Probability = %.5f\n", i, dequantized_values<i>); // If the probability of the current category is greater than 0.5, check whether it is greater than the maximum probability found if(dequantized_values<i> > 0.5) { // Update the maximum probability and the corresponding category numberif(probability < dequantized_values<i>) { probability = dequantized_values<i>; out_num = i; } } } return ret; } ``` ### Convert output value to probability```c++ void applySoftmax(float* data, int size) { // Find the maximum value in the array for numerical stabilization softmax calculationfloat max_val = data[0]; for (int i = 1; i < size; ++i) { if (data<i> > max_val) { max_val = data<i>; } } // Calculate the value of exp(xi - max_val) for each element and accumulate it for normalizationfloat sum_exp = 0; for (int i = 0; i < size; ++i) { data<i> = exp(data<i> - max_val); // Perform exponential operation on each value after subtracting the maximum value to prevent underflow caused by exponential operationsum_exp += data<i>; // Accumulate all exponential values for subsequent normalization} // Normalization step, convert all exp values into probability distribution (the sum of all probabilities is 1) for (int i = 0; i < size; ++i) { data<i> /= sum_exp; // Divide each element by the sum to get the probability value} } ``` ## Code running effect<div id="player_68fd412d539aec90db007434d4d9208e1" class="localvideo-box"></div><div class="localvideo-title">VLC media player 2024-05-25 16-54-12_Trim_Trim</div>5, then check whether it is greater than the maximum probability foundif(dequantized_values<i> > 0.5) { // Update the maximum probability and the corresponding category numberif(probability < dequantized_values<i>) { probability = dequantized_values<i>; out_num = i; } } } return ret; } ``` ### Output value converted to probability```c++ void applySoftmax(float* data, int size) { // Find the maximum value in the array for numerical stabilization softmax calculationfloat max_val = data[0]; for (int i = 1; i < size; ++i) { if (data<i> > max_val) { max_val = data<i>; } } // Calculate the value of exp(xi - max_val) for each element and accumulate it for normalizationfloat sum_exp = 0; for (int i = 0; i < size; ++i) { data<i> = exp(data<i> - max_val); // Subtract the maximum value from each value and perform exponential operation to prevent underflow caused by exponential operation sum_exp += data<i>; // Accumulate all exponential values for subsequent normalization} // Normalization step, convert all exp values into probability distribution (the sum of all probabilities is 1) for (int i = 0; i < size; ++i) { data<i> /= sum_exp; // Divide each element by the sum to get the probability value} } ``` ## Code running effect<div id="player_68fd412d539aec90db007434d4d9208e2" class="localvideo-box"></div><div class="localvideo-title">VLC media player 2024-05-25 16-54-12_Trim_Trim</div>5, then check whether it is greater than the maximum probability foundif(dequantized_values<i> > 0.5) { // Update the maximum probability and the corresponding category numberif(probability < dequantized_values<i>) { probability = dequantized_values<i>; out_num = i; } } } return ret; } ``` ### Output value converted to probability```c++ void applySoftmax(float* data, int size) { // Find the maximum value in the array for numerical stabilization softmax calculationfloat max_val = data[0]; for (int i = 1; i < size; ++i) { if (data<i> > max_val) { max_val = data<i>; } } // Calculate the value of exp(xi - max_val) for each element and accumulate it for normalizationfloat sum_exp = 0; for (int i = 0; i < size; ++i) { data<i> = exp(data<i> - max_val); // Subtract the maximum value from each value and perform exponential operation to prevent underflow caused by exponential operation sum_exp += data<i>; // Accumulate all exponential values for subsequent normalization} // Normalization step, convert all exp values into probability distribution (the sum of all probabilities is 1) for (int i = 0; i < size; ++i) { data<i> /= sum_exp; // Divide each element by the sum to get the probability value} } ``` ## Code running effect<div id="player_68fd412d539aec90db007434d4d9208e3" class="localvideo-box"></div><div class="localvideo-title">VLC media player 2024-05-25 16-54-12_Trim_Trim</div><br/>

## Result analysis: This model only has a good recognition rate for numbers with obvious features such as 6, 0, and 7. The analysis is probably because the framework of the neural network is too simple, and only three layers of convolution are used during training (see the previous article for specific training code). The neural network framework can be replaced with a mature structure such as MobileNetv3 (only conjecture and not practiced). ## Code sharing link: https://pan.baidu.com/s/1aSX1x_t9iWn5hv4Raj2_ow?pwd=ihrk Extraction code: ihrk</cv::vec4i></std::vector>

nmg

Did you write the image filtering code? If it is easy to arrange, can you show it?

"The filtering operation will traverse the entire image and reduce the frame rate." What impact will this have?

luyism

When copying image data to tensor memory space, 16-byte alignment is required. Copying the entire 28-byte row of image data directly into it does not meet this condition. This may be the reason that affects the recognition effect.

天空中

luyism posted on 2024-5-26 13:32 When copying image data to tensor memory space, 16-byte alignment is required. Copying the entire 28-byte row of image data directly into it does not meet the requirement...

Ok, I'll edit it, thanks.

#AIChallengeCampTerminal#RV1106HandwrittenDigitRecognitionDeployment [Copy link]

Latest reply

Comments