Fisheye camera parking space line detection solution for automatic parking technology

This article introduces the use of fisheye cameras for parking line detection. Looking at the current research on parking functions in the field of smart cars, from L2-level APA automatic parking to L2+ RPA remote control parking to L3-level HPA self-learning parking and finally AVP autonomous parking, each of which requires parking line detection. Through this article, let the editor take you to learn the relevant technologies and methods of parking line detection.

1

introduction

I have recently switched from panoramic camera perception to surround view perception. I feel that there is little difference in the implementation of pedestrian, vehicle and other target detection and semantic segmentation of road information. However, I have been struggling with how to do the parking line detection task for a long time. I have read all kinds of articles on deep learning for parking line detection in recent years. Here I will summarize and share some conventional methods.

2

What are the characteristics of parking space lines?

The ideal parking space line consists of four corner points and four lines, as shown in the figure below: the red one is called the entrance line, the two on the left and right are called the separate lines, and the purple line at the bottom is generally not very useful and can be used as a boundary.

There are usually three detection methods for targets of this shape:

a. The straight line-based method detects parking spaces by finding two segmentation lines and entrance lines. It appears in some systems that use traditional image processing algorithms to detect parking space lines. It uses operators such as Sobel and Canny for edge detection and combines Hough transform with geometric features to obtain potential parking space boundary lines. However, such traditional algorithms are easily affected by environmental factors such as lighting conditions, line wear, and ground shadows, and their performance lacks robustness.

b. The method based on the marked point is to detect the intersection of the entrance line and the two dividing lines, and then detect the parking space by combining the corner point coordinates. Traditional image processing algorithms provide a number of manually designed corner point detectors, such as Harris corner point detection, Shi-Tomasi corner point detection, FAST corner point detection, etc. If such methods are used, the robustness problem of the above-mentioned Hough line detection will still occur, so some scholars use the two intersection areas of the entrance line and the dividing line of the parking space line as the detection target, as shown in the following figure:

c. The segmentation-based method is to classify vehicles, free spaces, parking space signs and other objects pixel by pixel. This converts the parking space line detection problem into a semantic segmentation problem, which is similar to the lane line detection task in forward vision perception. However, the semantic segmentation task requires a series of complex post-processing to obtain relatively accurate parking spaces, which cannot meet the real-time requirements of the embedded end in terms of time consumption.

3

What are the forms of parking space lines?

There are three main types of parking lines: vertical, horizontal, and inclined. However, when doing parking line classification or program post-processing, you will encounter various structures, such as:

For example, some roadside parking spaces often use a curb instead of a dividing line; the entrance line and dividing line of some parking lots are separated; the color of the entire parking space is different from the surrounding area, but there is no parking line~~~ and so on.

4

How to get started learning parking space line detection?

In the perception module of autonomous driving, it is nothing more than the environmental perception of forward vision, perimeter vision, and surround vision. Using deep learning to extract features for classification is an essential technology. If you want to make a parking space line detection demo in the most convenient way, you need to do two preparations:

What kind of network is used for detection tasks?

What kind of data is used for training and verification?

In recent years, many methods for parking space line detection using deep learning have been open sourced. Here are a few of them (in no particular order):

"Attentional Graph Neural Network for Parking-slot Detection": This paper is divided into three stages: graph feature encoding, graph feature aggregation, and entry line identification. The graph neural network is used to aggregate the proximity information between the marked points on the spliced ​​bird's-eye view map to perform parking space line detection, solving the problem of redundant post-processing steps after independent detection of conventional marked points.

"Context-Based Parking Slot Detection With a Realistic Dataset": This paper is somewhat similar to the two-stage network of Faser RCNN, which is rough positioning + fine tuning. It first identifies whether there is a parking space in the PCR module, and then accurately locates the rotated BBox through the PSD module.

"DMPR-PS A Novel Approach For Parking-slot Detection Using Directional Marking-point regression" This paper detects directional marking points to obtain the intersection area of ​​all entrance lines and dividing lines in a spliced ​​bird's-eye view, and then filters and selects them to classify the parking space types.

"Vacant Parking Slot Detection in the Around View Image Based on Deep Learning": This paper is divided into two modules, parking line detection and parking space occupancy classification. The parking line detection module uses YOLOv3 to detect and classify the head area of ​​the entire parking slot to obtain the parking space type, eliminating the need to detect only two intersections and then obtain the parking space type through post-processing.

“PSDet: EfficientandUniversalParkingSlotDetection”: This paper also detects the intersections of parking lines. The difference is that it compares several forms of intersection feature descriptors. The circular feature descriptor is used to extract features within the intersection range, which can better identify the intersection area of ​​parking lines.

There are many related articles, so I won’t list them one by one. Friends who need related articles can contact me privately.

After talking about the network structure, let's take a look at the open source parking space line datasets. Currently, there are only three:

ps2.0

PSV

WoodScape

ps2.0 is a set of data stitched together from four fisheye cameras. All the data belong to parking scenes. The training set has about 10,000 images, the test set has less than 5,000 images, and the labels are in mat format. The data is shown in the figure below:

PSV is similar to the ps2.0 dataset, which is made from images stitched together by four fisheye cameras. The data volume is relatively small, with 2550 images in the training set, 425 images in the validation set, and 1274 images in the test set. The labels are semantic segmentation maps. The data is shown in the figure below:

The WoodScape dataset is more inclined to the independent environmental perception of each fisheye. The labels include about 10,000 external frame labels, about 10,000 semantic labels, about 10,000 depth maps, calibration files of 4 cameras, etc. It is suitable for 9 tasks such as segmentation, depth estimation, 3D bounding box detection, and spot detection. The data is shown in the figure below:

5

How to improve parking line detection?

The results of training based on open source datasets are often not applicable to one's own scenarios. It is necessary to collect and label small batches of corresponding scenarios to fine-tune the network in order to improve the detection effect of parking lines.

In this process we will encounter many problems, such as how to collect data, how much data to collect, and which scenarios to collect?

In fact, the most troublesome thing is the problem of dimensional division. Although there are only three types of parking space lines: vertical, horizontal, and inclined, there are many external environmental factors. For example:

The difference in the color of the parking space lines;

Different site materials;

Differences in light intensity;

Differences in weather conditions;

The difference in ground shadows;

Different wear and tear on parking spaces;

When formulating the collection standards, you must consider them carefully and thoroughly, and ensure a relatively balanced collection quantity for each type of data, otherwise the effect will be poor in some cases. The following figure is a dimension reference found in an article, and it should be OK to do a demo:

In addition to piling up data to improve performance, we can also consider the input and output forms of the network. Is the input a spliced ​​image of four fisheyes or the original data of a single fisheye?

Should the output be in the form of four key points of the parking space line, or in the form of two intersecting areas at the entrance, or in the form of a semantic graph? I am still considering this and have no good suggestions yet.

6

How to make it more like a product?

There is still a long way to go to turn the demo into a product. It is not enough to just focus on porting the prototype developed model to the embedded end and running it. Starting from the early stage of opening up the data link, we should pay attention to the RAW data type, ISP parameter adjustment, algorithm module design, hardware acceleration unit allocation, etc.; then to the data collection method, the number of annotation forms, how the tool chain can provide efficient iterative verification, as well as code quality review, performance evaluation between models, and a series of other work. The road is long and arduous, but I will keep exploring!