Kyungjae Lee
2025.05.02
[NAACL 2025 Best Paper Award] BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models
Image 1. LG AI Research's BiGGen Bench Study Selected as Best Paper at NAACL 2025[1]In the era of AI, the importance of language models continues to grow. Reflecting this, a wide range of language models have been introduced, accompanied by the development of numerous benchmarks to evaluate their capabilities. However, many of these benchmarks rely on abstract criteria such as preference, helpfulness, and harmlessness—making it difficult to distinguish models with high granularity and reliability[2,3,4].
Faced with this limitation, we posed a critical question “How can we identify the language model that best fits our needs and offers the highest utility?” To address this, LG AI Research’s Super Intelligence Lab partnered with Professor Minjoon Seo’s research team at KAIST to develop BiGGen Bench, a new benchmark for evaluating generative AI models. The project was a global collaboration involving researchers from Yonsei University, Carnegie Mellon University, Cornell University, MIT, the University of Washington, and the University of Illinois. To create BiGGen Bench, we defined nine core competencies of language models and 77 detailed task types, designing a total of 775 prompts and corresponding evaluation rubrics.
The study titled “The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models”[1] was presented at NAACL 2025, one of the most prestigious conferences in natural language processing, and was awarded the Best Paper Award, an honor given to only one paper among more than 2,000 submissions.
This recognition carries substantial significance. Of the 2,000+ papers submitted to NAACL 2025, roughly 1,400 were accepted—and only one received the Best Paper distinction. The selection of our work underscores the research value and practical impact of BiGGen Bench. This award joins the legacy of past NAACL Best Paper recipients such as ELMo (2018) and BERT (2019)—two groundbreaking studies in the history of language models.
Using BiGGen Bench, we also evaluated EXAONE 3.5, LG AI Research’s LLM released in December. Excluding reasoning models, EXAONE 3.5 demonstrated top-tier performance among recent non-thinking models, achieving an average score of 4.189. The recognition of BiGGen Bench by a top-tier academic conference and EXAONE 3.5's strong performance on the benchmark further validate our progress and commitment to excellence.
The study titled “The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models”[1] was presented at NAACL 2025, one of the most prestigious conferences in natural language processing, and was awarded the Best Paper Award, an honor given to only one paper among more than 2,000 submissions.
This recognition carries substantial significance. Of the 2,000+ papers submitted to NAACL 2025, roughly 1,400 were accepted—and only one received the Best Paper distinction. The selection of our work underscores the research value and practical impact of BiGGen Bench. This award joins the legacy of past NAACL Best Paper recipients such as ELMo (2018) and BERT (2019)—two groundbreaking studies in the history of language models.
Using BiGGen Bench, we also evaluated EXAONE 3.5, LG AI Research’s LLM released in December. Excluding reasoning models, EXAONE 3.5 demonstrated top-tier performance among recent non-thinking models, achieving an average score of 4.189. The recognition of BiGGen Bench by a top-tier academic conference and EXAONE 3.5's strong performance on the benchmark further validate our progress and commitment to excellence.
At NAACL 2025, we delivered an oral presentation on the BiGGen Bench study and had the opportunity to engage in valuable discussions with global researchers. This post takes a closer look at the motivations and methodology behind BiGGen Bench.
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models[1]
Motivation for the Study
Continued progress in large language models (LLMs) depends on our ability to precisely diagnose their capabilities, identify areas for improvement, and guide model enhancement through structured evaluation. While machine learning offers a range of evaluation metrics, the generative AI domain—especially long-form generation—still lacks a standardized, reliable framework for fair and fine-grained assessment.At LG AI Research, we are investigating self-improvement mechanisms in LLMs, guided by the following key research questions:
“How can we conduct high-resolution evaluation of language model outputs?
And can a language model evolve through structured feedback?”
And can a language model evolve through structured feedback?”
This led us to design and test a framework for automated self-improvement, built around a loop of generation, evaluation, and feedback. However, we soon confronted a fundamental trade-off in evaluation. Human experts can deliver highly reliable and nuanced assessments, but the process is labor-intensive and difficult to scale. LLM-based evaluators allow for scalable automation, but face challenges in accuracy, consistency, and interpretability.
To bridge this gap, we proposed a framework in which humans first define detailed evaluation rubrics across domains and task types, and then LLMs perform evaluations based on those guidelines. This hybrid approach allows for automated yet trustworthy evaluation and serves as the foundation for sustainable self-improvement in LLMs.
Key Contributions
As LLMs continue to take on more sophisticated tasks, the importance of granular and task-specific evaluation grows accordingly. Yet, many existing benchmarks have been criticized for their overly abstract criteria[2,3,4]. While some efforts have introduced domain-specific scoring rubrics[5], they often lack the resolution necessary for instance-level evaluation.
To overcome these challenges, we introduce BiGGen Bench, a new benchmark designed to comprehensively evaluate nine core abilities of LLMs. One of the key features of BiGGen Bench is its granular scoring logic, which is tailored to question types and designed to reflect subtle human judgment. For instance, when evaluating math problems, BiGGen Bench emphasizes logical reasoning and accuracy of computation, rather than relying solely on subjective helpfulness ratings.
Specifically, BiGGen Bench evaluates LLMs across the following nine competencies: Instruction Following, Grounding, Planning, Reasoning, Refinement, Safety, Theory of Mind, Tool Usage, Multilingualism.
These are assessed through 77 task types and 765 test items, offering a multidimensional, high-resolution evaluation framework. See Image 2 for an overview.
To overcome these challenges, we introduce BiGGen Bench, a new benchmark designed to comprehensively evaluate nine core abilities of LLMs. One of the key features of BiGGen Bench is its granular scoring logic, which is tailored to question types and designed to reflect subtle human judgment. For instance, when evaluating math problems, BiGGen Bench emphasizes logical reasoning and accuracy of computation, rather than relying solely on subjective helpfulness ratings.
Specifically, BiGGen Bench evaluates LLMs across the following nine competencies: Instruction Following, Grounding, Planning, Reasoning, Refinement, Safety, Theory of Mind, Tool Usage, Multilingualism.
These are assessed through 77 task types and 765 test items, offering a multidimensional, high-resolution evaluation framework. See Image 2 for an overview.
Image 2. The 77 Task Types Included in BiGGen Bench[1]
1) Components and Evaluation Methodology of BiGGen Bench
Each BiGGen Bench instance consists of four elements: System Message, Input Prompt, Reference Answer, and a Scoring Rubric. The System Message defines the evaluator’s role (e.g., that of a teacher or expert). The Input Prompt describes the specific task or query the language model must respond to. The Reference Answer provides an ideal response to help guide evaluation. The Scoring Rubric outlines explicit criteria for assigning scores on a 1–5 scale. When constructing BiGGen Bench, human experts were required to create detailed scoring rubrics for each instance, ensuring a high degree of evaluation granularity. Image 3 below illustrates an example involving a math problem, for which the human annotator designed fine-grained scoring criteria across a 5-point scale. Using this template, we prompted state-of-the-art LLM evaluators—such as GPT-4, Claude-3-Opus, and Prometheus-2—to provide both feedback and final scores according to the rubric. We then assessed alignment between human and model-generated scores, finding that GPT-4-Turbo achieved a Pearson correlation of 0.623, indicating strong consistency with human judgment.
Image 3. An example BiGGen Bench instance. Human annotators craft task-specific scoring rubrics using a 5-point scale[1]
2) BiGGen Bench Construction Protocol
BiGGen Bench was developed through the following four-stage process:
(a) Manual Instance Creation: A team of 18 co-authors created the core dataset by designing task instances aligned with nine capabilities. Native speakers contributed multilingual content in 10 languages to ensure linguistic diversity. Low-quality or redundant tasks were removed through iterative edits, such as grammar correction and difficulty adjustments. This resulted in 385 high-quality seed instances.
(b) Data Augmentation via In-context Learning: We augmented the seed data using GPT-4 to generate diverse variations of prompts. By providing in-context examples and encouraging semantic diversity in responses, we ensured high lexical variation. All augmented data was manually verified by human reviewers, yielding an additional 770 validated instances.
(c) Cross-verification: To ensure consistency, co-authors conducted cross-reviews of instances they did not originally create. Instances with inconsistent annotations or unclear rubrics were removed. This led to a finalized dataset of 765 instances.
(d) Human Judgment Collection: To validate the reliability of LLM-based automatic evaluations, we gathered independent human judgments on representative model responses. These assessments, particularly for multilingual tasks, played a crucial role in verifying the benchmark’s credibility and refining the evaluation process.
3) Analyzing the Performance of LLMs Using BiGGen Bench
Using BiGGen Bench, we evaluated 103 frontier LLMs and uncovered several key insights:
- Impact of Model Size: Performance generally increased with model size, consistent with previous findings that larger models tend to yield better outcomes through scaling.
Image 4. Performance trends of base LMs by model size[1]
- Pre-trained vs. Post-trained Models: Post-trained models significantly outperformed pre-trained models on Instruction Following tasks. However, the performance gap was less pronounced for Reasoning and Tool Usage, suggesting that post-training particularly enhances instruction adherence, but has limited effect on other abilities.
- Open-source vs. Proprietary Models: Proprietary models substantially outperformed open-source ones in Multilingualism, Reasoning, Theory of Mind, Planning, and Tool Usage. This indicates that open-source models may require further optimization in these critical areas.
Image 5. (Left) Performance gap between pre-trained and post-trained models. (Right) Gap between open-source and proprietary models[1]
Toward a New Evaluation Paradigm
In parallel with the BiGGen Bench initiative, LG AI Research open-sourced Prometheus-2, an evaluation model comparable to GPT-4 in quality[6,7]. This enables developers to tailor evaluation criteria to their specific domains and tasks—particularly for use cases involving sensitive data or custom workflows.
Looking ahead, we are advancing techniques that support faster, more reliable diagnosis of model capabilities, with a focus on domain-specific evaluation protocols for industrial applications. By establishing high-speed feedback loops, we aim to dramatically reduce the time required to iterate and improve LLMs.
Our evaluation framework goes beyond binary correctness, focusing instead on logical soundness, factuality, and completeness of reasoning steps. We believe this approach is essential to realizing the next generation of trustworthy, self-improving AI.
LG AI Research remains committed to shaping this new evaluation paradigm—where LLMs are automatically assessed and iteratively improved through feedback. We will continue to develop technologies that enhance evaluation accuracy and reliability, while expanding the benchmark’s adaptability across diverse domains and complex tasks.
Looking ahead, we are advancing techniques that support faster, more reliable diagnosis of model capabilities, with a focus on domain-specific evaluation protocols for industrial applications. By establishing high-speed feedback loops, we aim to dramatically reduce the time required to iterate and improve LLMs.
Our evaluation framework goes beyond binary correctness, focusing instead on logical soundness, factuality, and completeness of reasoning steps. We believe this approach is essential to realizing the next generation of trustworthy, self-improving AI.
LG AI Research remains committed to shaping this new evaluation paradigm—where LLMs are automatically assessed and iteratively improved through feedback. We will continue to develop technologies that enhance evaluation accuracy and reliability, while expanding the benchmark’s adaptability across diverse domains and complex tasks.
- 참고
[1] Kim, Seungone, et al. "The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models." In NAACL (2025).
[2] Zheng, Lianmin, et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
[3] Li, Xuechen, et al. "Alpacaeval: an Automatic Evaluator of Instruction-following Models." URL https://github. com/tatsu-lab/alpaca_eval (2023).
[4] Chan, Chi-Min, et al. "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate." The Twelfth International Conference on Learning Representations (2024).
[5] Ye, Seonghyeon, et al. "FLASK: Fine-grained Language Model Evaluation Based on Alignment Skill Sets." The Twelfth International Conference on Learning Representations (2024).
[6] Kim, Seungone, et al. "Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models." In EMNLP (2024).
[7] URL https://github.com/prometheus-eval/prometheus-eval