Multimodal Robotic Sensing System Based on Artificial Intelligence

The development of autonomous decision-making robotic systems that can mimic and surpass human sensing capabilities is crucial for agricultural, environmental, and safety monitoring. In particular, for on-site detection in some hazardous environments, robotic systems with multimodal sensing capabilities are needed to prevent threats to human health and safety. Skin is considered an ideal human-machine interface due to its high flexibility and adaptability and is widely used in multimodal robotic sensing systems.

According to MEMS Consulting, a research team at the California Institute of Technology has developed an artificial intelligence (AI)-driven multimodal robotic sensing system (M-Bot), which is based on a flexible electronic skin human-machine interface (H) and can perform rapid and ultra-sensitive on-site analysis of temperature, tactile pressure, and various hazardous chemicals. The research is expected to improve the perception capabilities of intelligent robots and pave the way for many new practical wearable and robotic applications. The relevant research results have been published in the journal Science Robotics.

M-Bot consists of two fully inkjet-printed flexible electronic skin patches, namely e-skin-R (connected to the robot) and e-skin-H (connected to human skin). e-skin-R consists of a high-performance multimodal physical and chemical sensor array that can be installed on the palm and fingers of the robot, and e-skin-H consists of four surface electromyography (sEMG) electrode arrays (channels) and a pair of electrical stimulation electrodes. Under the action of AI, multimodal physical and chemical sensing and electrical stimulation feedback control, e-skin-R and e-skin-H form a closed-loop human-machine interactive robot sensing system.

e-skin-R: All-inkjet printed multimodal array

e-skin-H: for AI-assisted human-computer interaction

Through precise control, M-Bot can achieve highly sensitive human-computer interaction. Its multimodal sensor data can be collected in real time through a portable multi-channel constant potential instrument, and wirelessly transmitted and displayed on a mobile phone application. M-Bot is also capable of object grasping, multi-point touch and chemical sensing. Multiple physical and chemical data can be recorded and automatically processed simultaneously without signal interference. The researchers pointed out that by further increasing the number and density of multimodal sensor arrays, more accurate and detailed data can be obtained from any object and surface.

Human-machine interactive sensing experiment based on M-Bot

M-Bot can also be further extended to robotic systems for risk warnings such as tracking objects, explosives, chemical threats and biological hazards, and provide interactive tactile and threat alert feedback. The researchers conducted relevant experiments and applied e-skin-R to a multimodal sensing robot boat (M-Boat), whose sensor array includes one, three chemical sensors, two (for boat propulsion and steering) and one (), for data collection, signal processing and, capable of continuous hazard analysis and autonomous leak tracking in seawater. The researchers pointed out that if necessary, more biosensors (such as pH and conductivity sensors) can be integrated into the e-skin-R, and more real-time calibration mechanisms for precise analysis can be introduced. In the future, the M-Boat system is also expected to be applied to the field to provide solutions for intelligent path planning and decision-making for automobiles.

e-skin-R technology can be applied to multi-modal sensing robotic boat (M-Boat)

M-Bot provides a deeper level of interactive cognitive capabilities for robot autonomous decision-making and greatly broadens the range of tasks that robots can perform. This technical strategy will play an important role in designing more advanced intelligent robot systems.

Paper information: https://doi.org/10.1126/sciroboTIcs.abn0495

Editor: Huang Fei