EMNLP Conference (2025)
Sanghyun Seo, Bumsoo Kang, Dahm Lee, Jaeheon Kim, Joongbo Shin, Euisoon Kim, Kijeong Jeon
Abstract
To effectively support users’ goal achievement in chat-LLM services, providing usercentered follow-up questions is essential. Existing studies have largely focused on enhancing information-seeking or topical relevance, often missing how follow-up questions could help satisfy users’ intrinsic needs and achieve conversational goals. To bridge this gap, we introduce FQ-Eval1 , a user-centered evaluation dataset designed for assessing follow-up question generation in chat-LLM services. FQEval incorporates realistic chat-LLM usage scenarios and five distinct human-aligned criteria, each reflecting user expectations of effective follow-up questions. Experimental results show that follow-up questions in FQ-Eval clearly capture these human-aligned dimensions, enabling robust, human-aligned evaluation of follow-up question generation for various models and services.