How to use big models on Xiaohongshu? Top conference authors are waiting for you to chat online

Yunzhong sent from Aofei Temple
Quantum Bit | Public Account QbitAI

Big models are leading a new round of research boom, with numerous innovative results emerging in both industry and academia.

Xiaohongshu's technical team has also been constantly exploring in this wave, and many of its research papers have been presented at top international conferences such as ICLR, ACL, CVPR, AAAI, SIGIR, and WWW.

What new opportunities and challenges has Xiaohongshu discovered at the intersection of big models and natural language processing?

What are the effective evaluation methods for large models? How can they be better integrated into application scenarios?

From 19:00 to 21:30 on June 27, [REDtech is here] the eleventh issue of "Little Red Book 2024 Large Model Frontier Paper Sharing" will be broadcast online!

REDtech specially invited the Xiaohongshu community search team to the live broadcast room, where they will share 6 large model research papers published by Xiaohongshu in 2024.

Feng Shaoxiong, head of Xiaohongshu's LTR ranking, joined hands with many top conference paper authors Li Yiwei, Wang Xinglin, Yuan Peiwen, Zhang Chao and others to discuss the latest large model decoding and distillation technology, large model evaluation methods, and the practical application of large models on the Xiaohongshu platform.

Book a live broadcast and the first authors of multiple papers will communicate with you online! You will gain the latest insights on large model technology, explore future development trends, and exchange ideas on how to use these cutting-edge technologies to enhance user experience and promote the intelligent development of the platform.

Event Agenda

01 Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning / Selected for ICLR 2024

An early stopping self-consistent method for high cost problem in multi-step reasoning of large models | Speaker: Li Yiwei

Self-Consistency (SC) has been a widely used decoding strategy in thought chain reasoning, which improves the performance of the model by generating multiple thought chains and taking the majority answer as the final answer. However, it is a high-cost method that requires multiple samplings of a preset size.

At ICLR 2024, Xiaohongshu proposed a simple and scalable sampling process, Early-Stopping Self-Consistency (ESC), which can significantly reduce the cost of SC without sacrificing performance. On this basis, the team further derived an ESC control scheme to dynamically select the performance-cost balance of different tasks and models. Experimental results on three mainstream reasoning tasks (mathematics, common sense, and symbolic reasoning) show that ESC significantly reduces the average number of sampling times in six benchmarks while almost maintaining the original performance.

Paper address: https://arxiv.org/abs/2401.10480

02 Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation / Selected for ACL 2024

Refining the coarse and retaining the fine: A fine-grained self-consistent approach to free-form generation tasks | Speaker: Wang Xinglin

Xiaohongshu proposed the Fine-Grained Self-Consistency (FSC) method in ACL 2024, which can significantly improve the performance of self-consistency methods in free-format generation tasks.

The team first conducted an experimental analysis and found that the shortcomings of existing self-consistent methods for free-form generation tasks come from the coarse-grained selection of common samples, which cannot effectively utilize the common knowledge between fine-grained fragments of different samples.

On this basis, the team proposed the FSC method based on large model self-fusion. Experiments have confirmed that it has achieved significantly better performance in code generation, summary generation and mathematical reasoning tasks while maintaining considerable consumption.

Paper address: https://github.com/WangXinglin/FSC

03 BatchEval: Towards Human-like Text Evaluation / Selected for ACL 2024, the field chair gave a full score and recommended the best paper

Towards human-level text evaluation | Speaker: Yuan Peiwen

Xiaohongshu proposed the BatchEval method in ACL 2024, which can achieve human-level text evaluation results with lower overhead.

The team first analyzed from a theoretical perspective that the shortcomings of existing text evaluation methods in terms of evaluation robustness come from the uneven distribution of evaluation scores, and the suboptimal performance in score integration comes from the lack of diversity in evaluation perspectives.

On this basis, inspired by the human evaluation process to establish a more comprehensive and diverse evaluation benchmark through sample comparison, BatchEval was proposed. Compared with several current state-of-the-art methods, BatchEval has achieved significantly better performance in both evaluation overhead and evaluation effect.

Paper address: https://arxiv.org/abs/2401.00437

04 Poor-Supervised Evaluation for SuperLLM via Mutual Consistency / Selected for ACL 2024

Achieving Superhuman Level Evaluation of Large Language Models in the Lack of Accurate Supervision Signals through Mutual Consistency | Speaker: Yuan Peiwen

Xiaohongshu proposed the PEEM method in ACL 2024, which can achieve accurate evaluation of large language models that surpass human level through mutual consistency between models.

The team first analyzed that the current rapid development trend of large language models will accelerate their gradual reaching or even surpassing human levels in many aspects. In this case, it will be difficult for humans to provide accurate evaluation signals.

To achieve capability assessment in this scenario, the team proposed the idea of ​​using mutual consistency between models as an evaluation signal, and deduced that when the evaluation samples are infinite, if there is an independence of prediction distribution between the reference model and the model to be evaluated, then the consistency with the reference model can be used as an accurate measure of the model's capability.

On this basis, the team proposed the PEEM method based on the EM algorithm. Experiments have confirmed that it can effectively alleviate the insufficiency of the above conditions in reality, thereby achieving accurate evaluation of large language models that surpass the human level.

Paper address: https://github.com/ypw0102/PEEM

05 Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data / Selected for AAAI 2024 Oral

Using negative samples to promote the distillation of large model reasoning ability | Speaker: Li Yiwei

Large language models (LLMs) have excellent performance on various reasoning tasks, but their black-box properties and large number of parameters hinder their widespread application in practice. In particular, LLMs sometimes produce incorrect reasoning chains when dealing with complex mathematical problems.

Traditional research methods only transfer knowledge from positive samples, while ignoring synthetic data with wrong answers. At AAAI 2024, the Xiaohongshu search algorithm team proposed an innovative framework, which first proposed and verified the value of negative samples in the model distillation process, and built a model specialization framework that, in addition to using positive samples, also makes full use of negative samples to refine the knowledge of LLM.

The framework consists of three sequential steps, including negative assisted training (NAT), negative calibration enhancement (NCE), and dynamic self-consistency (ASC), covering the full process from training to inference. A series of extensive experiments demonstrate the key role of negative data in LLM knowledge distillation.

Paper address: https://arxiv.org/abs/2312.12832

06 NoteLLM: A Retrievable Large Language Model for Note Recommendation / Selected for WWW 2024

Note content representation recommendation system based on large language model | Speaker: Zhang Chao

A large number of new notes are generated every day on the Xiaohongshu APP. How can we effectively recommend these new contents to interested users? Recommendation representation based on note content is a way to alleviate the cold start problem of notes and is also the basis for many downstream applications.

In recent years, large language models have attracted much attention due to their powerful generalization and text understanding capabilities. Therefore, Xiaohongshu hopes to use large language models to build a note content representation recommendation system to enhance the understanding of note content. The technical team will introduce recent work from two perspectives: generating enhanced representations and multimodal content representations.

Currently, the system has been applied to multiple business scenarios of Xiaohongshu and achieved significant benefits.

Paper address: https://arxiv.org/abs/2403.01744

How to watch the live broadcast

Live broadcast time : 19:00-21:30, June 27, 2024

Live broadcast platform : WeChat video account [Redtech], Bilibili, Douyin, and Redbook accounts with the same name for real-time live broadcast.

You are welcome to fill out the questionnaire to provide feedback on your concerns about the big model and interact in depth with the guests during the live broadcast.

Scan the QR code below to enter the live broadcast discussion group, and get the live broadcast link and broadcast reminder as soon as possible; you can get the carefully compiled [Paper PDF Collection] with one click, and have the opportunity to communicate directly with the paper authors!

Invite friends to book a live broadcast gift

recruitment

Xiaohongshu Community Search Team is recruiting for multiple positions. The team is responsible for optimizing Xiaohongshu search results and exploring cutting-edge technologies, and is committed to building China's largest life search engine. We look forward to your joining us! (Click "Read More" to learn more about recruitment positions)