PathPartner launches occupant monitoring system using 4D millimeter-wave radar and camera fusion technology

According to foreign media reports, product development and engineering expert PathPartner recently announced the development of OMS (occupant monitoring sensor) using 4D imaging radar technology and camera fusion technology, aiming to supplement the camera-based driver monitoring system. However, PathPartner's DMS (driver monitoring system) can combine the advantages of both and provide automakers with a single and low-cost component.

“Combining cameras and radars significantly improves accuracy in low-light, noisy environments,” said Vinay MK, VP of R&D at PathPartner. “PathPartner’s camera sensors come with near-infrared illumination, which provides reliable performance in varying lighting conditions.” When used as part of a driver or occupant monitoring system, PathPartner’s 4D imaging radar technology can detect subtle movements and even breathing inside the vehicle, such as a child sleeping under a blanket, or a pet.

(Image source: PathPartner)

Vinay MK also said: "The camera-based occupant monitoring system can detect whether the seat belt is used correctly and the body size of the occupant, which are important information for deploying airbags. The system monitors driver activities such as smoking, talking on the phone, eating, drinking, texting, distraction, sleeping and sudden illness. In addition, it can also detect certain objects such as empty baby seats, hanging coats, etc."

Vina added: “The 4D radar system can detect if someone is in the seat in no-light conditions, thus perfectly complementing the camera-based monitoring system. The 60GHz radar system has a high resolution and is able to detect the difference in breathing rate between an infant and an adult. Combining the camera and radar data can significantly improve accuracy in low-light conditions.”

Vinay said: "Even if a baby is covered with a blanket in a child seat, it can be detected by the 4D radar module alone. One of the important reasons for using 4D radar in DMS is to identify whether there are passengers in the vehicle when it is parked."

The National Safety Council reports that an average of 39 children under the age of 15 die each year from heat stroke after being left in a car. As a result, Congress enacted the Hot Cars Act in 2019, requiring the U.S. Department of Transportation (DOT) to mandate that all new vehicles be equipped with child detection systems, one type of which is the occupant monitoring system being developed by PathPartner.

To meet the needs of automakers, PathPartner DMS has been ported to South Korean semiconductor manufacturing company NextChip's Apache4, a preprocessor for vision systems in advanced driver assistance systems (ADAS). According to Nextchip, Apache4 contains a dedicated image processing accelerator subsystem and optimized software that can reduce computing requirements by 70%.

“The Apache4 processor is highly integrated and provides advanced image processing. Its advanced algorithms provide higher detection rates than traditional vector-based classifiers, making it a cost-effective DMS solution for the mainstream Euro NCAP (new car crash test) market,” said Vinay.

According to PathPartner, its DMS design revolves around multiple small convolutional neural network (CNN) models and is combined with an adaptive machine learning module. Through this hybrid architecture, its imaging radar technology uses Qualcomm's DSP (digital signal processing system) to process 60 frames of images per second provided by the camera with a smaller memory. Combining PathPartner's SOM (system-on-module, high-performance modular system) and smart camera reference design kit with its equipment engineering services can help equipment manufacturers meet imaging radar needs outside the automotive industry, including devices based on AI cameras and IoT (Internet of Things).

Driver and occupant monitoring systems are not only useful for autonomous vehicles, commercial vehicle fleet operators can also improve the safety of their fleets by monitoring drivers. The technology can also be used to monitor rear seat occupants to ensure that no people or pets are accidentally left in the vehicle.

PathPartner has also established a strategic partnership with a Tier 1 supplier that specializes in the development of advanced vision systems for mobility. PathPartner declined to name the company at this time. However, Vinay said the two companies are currently jointly developing camera- and radar-based occupant monitoring and driver monitoring ECUs to support the processing needs of the system.

PathPartner's strategic partners will integrate its occupant monitoring system (OMS) and DMS algorithms on automotive-grade ECUs. To reduce latency, PathPartner does not transmit data to the cloud for processing, but instead applies edge computing to occupant sensors. For the automotive industry, this approach is safer and reduces reliance on cellular network data transmission, which is not always reliable enough for safety-critical automotive systems.