Sign up to save your podcasts
Or
Share CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
🤗 Paper Upvotes: 50 | Categories: cs.IR, cs.CL
Title:
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
Authors:
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen, Zhicheng Dou
Arxiv:
http://arxiv.org/abs/2410.23090v1
Abstract:
Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing large language models (LLMs) through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses on single-turn RAG, leaving a significant gap in addressing the complexities of multi-turn conversations found in real-world applications. To bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess RAG systems in realistic multi-turn conversational settings. CORAL includes diverse information-seeking conversations automatically derived from Wikipedia and tackles key challenges such as open-domain coverage, knowledge intensity, free-form responses, and topic shifts. It supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. We propose a unified framework to standardize various conversational RAG methods and conduct a comprehensive evaluation of these methods on CORAL, demonstrating substantial opportunities for improving existing approaches.
View all episodes
By Jingwen Liang, Gengyu Wang🤗 Paper Upvotes: 50 | Categories: cs.IR, cs.CL
Title:
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
Authors:
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen, Zhicheng Dou
Arxiv:
http://arxiv.org/abs/2410.23090v1
Abstract:
Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing large language models (LLMs) through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses on single-turn RAG, leaving a significant gap in addressing the complexities of multi-turn conversations found in real-world applications. To bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess RAG systems in realistic multi-turn conversational settings. CORAL includes diverse information-seeking conversations automatically derived from Wikipedia and tackles key challenges such as open-domain coverage, knowledge intensity, free-form responses, and topic shifts. It supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. We propose a unified framework to standardize various conversational RAG methods and conduct a comprehensive evaluation of these methods on CORAL, demonstrating substantial opportunities for improving existing approaches.