Multimodal celebrities all open Xiaohongshu

In the real world, humans use a combination of vision, hearing, touch, smell and other senses to access and understand the world. The information we obtain through different senses is naturally in a "multimodal" form.

In this sense, the development of artificial intelligence is a process of approaching human intelligence. Multimodal learning is an inevitable development direction.

Multimodal learning brings new application scenarios

In an era of explosive Internet information and increasingly diverse forms, the proportion of text, pictures and short videos in Internet content continues to increase. A single modality often cannot provide a complete description of information such as text, pictures and videos. At the same time, the understanding of content in the application field runs through the entire search and recommendation system.

Just as we are accustomed to using "image recognition" instead of text search on shopping apps; in the field of smart homes, voice interaction and gesture interaction are becoming popular trends; communication with smart robots is no longer a mechanical text dialogue, but has entered a deeper stage of voice understanding and image understanding.

We need to understand content from multiple granularities. How to integrate feature information from multiple modalities has gradually become a new challenge that exists in many fields. Therefore, it is a consensus in academia and industry today to focus on the development of multimodal technology.

Today’s Challenges

Although multimodality has been around since the 1970s and has undergone decades of development, it still faces many challenges in the current industrial implementation, resulting in some scenarios being in a "pseudo-multimodal" application state, which affects the user experience.

• The “semantic gap” still exists;

• How to master large amounts of matching multimodal data;

• uncertainty in multimodal information;

• Fine alignment between different modalities;

• Effective architectures for multimodal pre-training, and more.

So, now that academia and industry are once again focusing on multimodality, what are they discussing?

Expert interpretation is waiting for you

At 19:00 on April 20, the first episode of the live broadcast program "REDtech is here" produced by the Xiaohongshu technical team will focus on the topic of multimodality.

In the first half of the issue, the Xiaohongshu technical team invited Xie Weidi, associate professor and doctoral supervisor from the School of Electronic Information and Electrical Engineering of Shanghai Jiao Tong University, Liu Si, professor and doctoral supervisor from the Beijing University of Aeronautics and Astronautics, and Gao Shenghua, associate professor and doctoral supervisor from the School of Information of ShanghaiTech University to share their research on multimodal content understanding.

The second half of the live broadcast, which is expected to be held on April 27, will focus on multimodal understanding and creation. He Ran, a researcher at the Institute of Automation, Chinese Academy of Sciences, Zhou Xiaowei, a "Hundred Talents Program" researcher and doctoral supervisor at Zhejiang University, and Zhu Chaolin, a lecturer at the ReLER Laboratory of the University of Technology Sydney, will bring the latest research results on multimodality from the academic community.

Scholars from the above-mentioned universities will share topics such as " Cross-modal image content understanding and video generation ", " Language-guided visual localization ", " Multimodal visual content generation ", " Multimodal retrieval, localization and generation methods ", " Convenient three-dimensional digitization technology ", and " Technology and application of self-supervised learning in multimodal content understanding ". You are welcome to interact and ask questions in the live broadcast room!