University of Michigan develops real-time 3D motion tracking system to replace lidar and self-driving cameras

According to foreign media reports, the University of Michigan has developed a new real-time 3D motion tracking system that can replace imaging system technologies such as lidar and cameras in autonomous driving applications.

(Image source: University of Michigan)

The technology combines transparent light detectors with advanced neural networks and is expected to be used in automated manufacturing, biomedical imaging and autonomous driving. Zhaohui Zhong, an associate professor of electrical and computer engineering at the University of Michigan, and his team developed a transparent, nanoscale, highly sensitive graphene photodetector, and the imaging system takes advantage of this detector.

"The in-depth combination of graphene nanodevices and machine learning algorithms can bring opportunities in science and technology," said Dehui Zhang, a doctoral student in electrical and computer engineering. "Compared with several other solutions, our system has the advantages of high computational efficiency, fast tracking speed, compact hardware and lower cost."

The graphene photodetectors are tuned to absorb about 10% of the light that comes into contact with them. Because graphene is so sensitive to light, it is enough to produce images that can be reconstructed through computational imaging. These photodetectors are stacked together to form a compact system, with each layer focusing on a different focal plane, thus achieving 3D imaging.

In addition to 3D imaging, the team also deals with real-time motion tracking. To do this, they need a way to determine the position and orientation of the object being tracked. Typical approaches include lidar systems and light field cameras, but both have significant limitations, the researchers said.

According to the University of Michigan, some people use metamaterials or multiple cameras, but hardware alone cannot achieve the desired effect without the introduction of deep learning algorithms. Zhen Xu, a doctoral student in electrical and computer engineering, created the optical device and worked with the team to enable the neural network to interpret the position information.

The neural network is trained to search for specific objects throughout a scene and then focus only on the object of interest. "It takes time to train the neural network, but once it's trained, it can give an answer within milliseconds when the camera sees a scene," said Ted Norris, professor of electrical and computer engineering and the project leader.

The team successfully tracked a beam of light and tracked a ladybug using two stacks of 4×4 pixel graphene photodetector arrays. In addition, the researchers also proved that the technology is scalable. They believe that for some practical applications, only 4,000 pixels are needed, while for more applications, only a 400×600 pixel array is needed. Although the imaging system technology can be used with other materials, one advantage of graphene is that it does not require artificial lighting and is environmentally friendly.