Seungdong Yoa
2026.11.05
[ICLR2026] Beyond Static Benchmarks: An Agent-Centered Dynamic Assessment Protocol (ATAD) for Evaluating LLM Reasoning Capabilities
Large language models (LLMs) have been advancing rapidly in recent years. On existing benchmarks such as MMLU[2] and GSM8K[3], models have already demonstrated performance that surpasses human capabilities. However, these fixed benchmarks are increasingly revealing their limitations in evaluating a model’s actual reasoning ability. Furthermore, concerns continue to be raised regarding publicly available benchmarks, such as data contamination in large-scale pre-training datasets and overfitting to specific evaluation formats. The paper also identifies performance plateauing, data contamination, and the limited lifespan of benchmarks as major issues with static evaluation.
To address these issues, LG AI Research proposed ATAD (Agent-Centric Text Anomaly Detection), a dynamic evaluation protocol that moves beyond the conventional approach of evaluating fixed datasets, allowing the evaluation process itself to evolve in accordance with the model’s capabilities. ATAD consists of three agents, Teacher, Orchestrator, and Student, that repeatedly generate, validate, and solve problems. When the Student solves a problem, the Teacher generates a more difficult one, while the Orchestrator continuously validates the problem’s clarity, logical consistency, and fairness throughout the process to prevent problems from becoming ambiguous as the difficulty increases. In other words, the core of ATAD lies in the fact that it presents a protocol that does not stop at building a new benchmark dataset once, but rather continuously adjusts the difficulty and quality of evaluation problems in line with the model’s progress.
The related research was accepted and presented at the ICLR 2026 main conference under the title “From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning”. ICLR (International Conference on Learning Representations) is a world-renowned international conference where the latest research findings in the field of machine learning, with a focus on representation learning and deep learning, are shared. ICLR 2026 was held in Rio de Janeiro, Brazil, from April 23 to 27.
1. Limitations of existing static benchmarks and new motivation
The static benchmarks currently widely used for evaluating LLMs have the following structural limitations.
● Performance Saturation: The latest LLMs have already surpassed human-level performance on static benchmarks, leading to a loss of ability in distinguishing performance differences between models.
● Data Contamination: Evaluation data is included in the large-scale pre-training process, causing models to memorize it and resulting in inflated leaderboard scores.
● Format Overfitting: Model developers tune the system to specific formats within the dataset, creating a feedback loop where scores rise without any actual improvement in general-purpose reasoning capabilities.
● Short Lifespan: Once a benchmark is conquered, building a new one requires an enormous amount of time and resources, making sustainable evaluation difficult.
Due to these issues, there has been a growing need for dynamic evaluation, a method that prevents data corruption and allows the model to continuously evolve in line with advancements in the model itself.
However, while existing dynamic benchmarks have been limited to simply generating or scaling problems, ATAD distinguishes itself by introducing an “agent-centric three-party interaction structure (Teacher-Orchestrator-Student).” In particular, the Orchestrator is solely responsible for validation, preventing a loss of clarity and an increase in ambiguity that can occur as the difficulty level rises, and by identifying the points where the student agent fails, enabling “Failure-Driven Sample Finalization”, it has created a high-quality benchmark protocol that more precisely reveals the model’s actual blind spots.
In particular, when creating problems in the field of Text Anomaly Detection, researchers consistently face a major challenge: the “trade-off between clarity and difficulty.” This is because, unlike in mathematics or science, language does not have clear mathematical formulas or fixed answers; consequently, attempting to increase the difficulty often results in ambiguous correct answers, while making the correct answer too clear makes the problem too easy, causing it to lose its discriminative power.
Figure 1. Comparison of text anomaly detection issues between existing benchmarks and ATAD[1]
For example, existing benchmarks often insert a completely out-of-context sentence, such as “Oil prices fell following an OPEC announcement,” into a passage with “A tennis champion withdrew due to an injury.” While the correct answer is obvious, this is a trivial problem for state-of-the-art LLMs.
ATAD, on the other hand, is far more sophisticated. Within the context of “AI is revolutionizing healthcare,” it induces a subtle shift in topic, such as listing technical benefits like treatment plan optimization or real-time vital sign monitoring, only to suddenly raise ethical concerns about AI. This creates a challenging problem that requires advanced linguistic and logical reasoning skills, while still maintaining a clear basis for the correct answer.
2. ATAD Protocol Structure and 7 Text Anomaly Detection Tasks
ATAD addresses this problem by introducing a protocol in which three AI agents interact, rather than using a fixed dataset. Unlike mathematics or code, natural language lacks clear formulas, making text anomaly detection a rigorous stress test to validate whether it is possible to increase the difficulty while maintaining clarity.
● Teacher Agent (Questioner): Generates basic problems; if a Student answers a problem correctly, it generates increasingly difficult variations based on the analysis.
● Orchestrator Agent (Validator): Strictly validates the logical consistency, clarity, and fairness of the generated problems to prevent ambiguous or inappropriate adversarial difficulty increases.
● Student Agent (Solver): Solves the validated problems. When a Student fails a problem, that problem is confirmed as the final benchmark item (Failure-Driven Sample Finalization).
Figure 2. ATAD’s Teacher-Orchestrator-Student-based dynamic evaluation protocol[1]
ATAD designed a total of seven anomaly detection tasks (T1-T7) to prevent superficial pattern matching and evaluate cross-sentence logical reasoning.
Representative tasks include Sentence Context Anomaly (T1), which finds semantic deviations in context; Referential Ambiguity (T5), which finds ambiguity in demonstrators or pronouns; and Tone/Style Violation (T7), which finds inconsistencies in style and tone. Each task validates the model's various reasoning capabilities from multiple angles.
Figure 3. Examples of 7 text-based anomaly detection tasks defined by ATAD[1]
Representative tasks include Sentence Context Anomaly (T1), which finds semantic deviations in context; Referential Ambiguity (T5), which finds ambiguity in demonstrators or pronouns; and Tone/Style Violation (T7), which finds inconsistencies in style and tone. Each task validates the model's various reasoning capabilities from multiple angles.
Figure 3. Examples of 7 text-based anomaly detection tasks defined by ATAD[1]
3. Experiment Results Analysis: Difficulty expansion and proof of the Orchestrator's quality control
When the benchmarks generated by ATAD were evaluated against various LLMs, including GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0-Flash, some interesting findings were discovered.
First, the performance patterns of LLMs varied across tasks. Even models that showed relatively high performance on average performed worse than other models on specific tasks, and no single model consistently demonstrated the highest performance across all seven tasks. This demonstrates that each model has its own unique strengths and weaknesses (blind spots) in different reasoning abilities, such as contextual understanding, logical connections, reasoning of the target of instructions, and stylistic judgment. In the actual experiments, performance patterns also varied significantly among models depending on the type of anomaly.
Table 1. Performance comparison by LLM across 7 tasks of the ATAD benchmark[1]
First, the performance patterns of LLMs varied across tasks. Even models that showed relatively high performance on average performed worse than other models on specific tasks, and no single model consistently demonstrated the highest performance across all seven tasks. This demonstrates that each model has its own unique strengths and weaknesses (blind spots) in different reasoning abilities, such as contextual understanding, logical connections, reasoning of the target of instructions, and stylistic judgment. In the actual experiments, performance patterns also varied significantly among models depending on the type of anomaly.
Table 1. Performance comparison by LLM across 7 tasks of the ATAD benchmark[1]
Second, across all models, the accuracy rate for the final problems dropped significantly compared to the base problems. This indicates that the ATAD protocol did not stop at problems the Student could already solve, but instead generated actually more difficult problems through iterative interaction between the Teacher and the Student. Even when GPT-4o, Gemini-2.0-Flash, Claude-3.5-Sonnet, and LLaMA-3.3-70B were used for benchmark generation, evaluation accuracy consistently decreased as the models progressed from base to final, and the paper observed an overall performance decline of approximately 21.3-33.9%.
Table 2. Comparison of LLM performance on the base problem and the final problem generated through ATAD[1]
Table 3. Comparison of problem difficulty and quality based on whether the Orchestrator is applied[1]
Table 2. Comparison of LLM performance on the base problem and the final problem generated through ATAD[1]
Third, the most significant finding emerged from the Orchestrator’s quality control role. In the experiment, when problems were generated solely through competition between the Teacher and Student, excluding the Orchestrator, GPT-4o’s evaluation accuracy was 68.29%, which was lower than the 72.43% achieved when the Orchestrator was involved. If one looks solely at the model’s accuracy rate, the problems without the Orchestrator might appear to be more difficult; that is, a better benchmark.
However, when the quality of the problems themselves is evaluated, a different result emerges. When GPT-4o was used as the reviewer and Orchestrator was applied, the problems’ Validity increased from 4.30 to 4.85, Coherence from 3.71 to 4.74, and Fairness from 3.20 to 4.65. The Approval Rate, which is the percentage of problems deemed suitable for use as benchmarks, also increased significantly from 38.14% to 87.14%. The same trend was observed when using Claude-3.5-Sonnet as the reviewer.
Table 3. Comparison of problem difficulty and quality based on whether the Orchestrator is applied[1]
In other words, the fact that a model makes many mistakes does not necessarily mean it is a good benchmark. This is because the model’s accuracy may be low even if the correct answer is ambiguous, multiple answers are possible, or the problem itself contains logical flaws. In such cases, poor performance may not reflect the model’s limitations in reasoning but rather an artificial difficulty caused by a poorly designed problem.
In ATAD, the Orchestrator is responsible for distinguishing between these two scenarios. Even if the Teacher generates more difficult problems, the Orchestrator rejects them and prompts the Teacher to generate new ones if logical consistency, clarity, or fairness are not maintained. Therefore, ATAD does not simply aim to lower the model’s accuracy; rather, it aims to increase the difficulty of reasoning while maintaining the criteria for a “good” problem. The paper also confirmed that the low accuracy observed in cases without the Orchestrator may stem from ambiguous or ill-posed problems rather than actual reasoning difficulty.
This result is directly linked to the trade-off between clarity and difficulty described earlier. The Teacher-Student competition increases the difficulty of the problems, while the Orchestrator controls the process to ensure that clarity is not compromised. By separating these two roles, ATAD makes it possible to simultaneously generate both difficult and high-quality problems in natural language tasks.
Figure 4. Example of problem improvement through Orchestrator feedback[1]
Figure 4. Example of problem improvement through Orchestrator feedback[1]
This role can also be observed in real-world examples. In Figure 2, the initially generated problem contained sentences that were all somewhat related to the topic, making it difficult to determine which sentence was a clear anomaly. Rather than accepting this as a high-difficulty problem, the Orchestrator rejected it, judging that the anomaly was not sufficiently clear. Subsequently, after receiving feedback, the Teacher regenerated the problem, improving it into one that naturally fits within the overall context while still containing a clear semantic error.
As such, Quality Control in ATAD is not a mechanism designed to lower difficulty. Rather, it is a process intended to eliminate incorrect difficulty levels stemming from ambiguity and retain only those that truly require actual reasoning ability.
4. Not a new dataset, but an evaluation protocol that evolves alongside the model
The change that ATAD ultimately proposes goes beyond simply adding a new Text Anomaly Detection dataset. In existing static benchmarks, once an evaluation problem is created, the same problem is used repeatedly. Therefore, as models improve and begin to achieve high performance on that benchmark, it becomes increasingly difficult to distinguish differences in ability among models over time.
In contrast, ATAD defines a benchmark not as a fixed set of data, but as a protocol through which the Teacher-Orchestrator-Student interact. Currently, when the Student solves a problem, the Teacher generates a more difficult problem, and the Orchestrator validates that the problem remains clear and fair. Therefore, even if a more powerful model emerges to take on the role of the Student, the difficulty of the evaluation can be raised again to match that model’s capabilities.
Figure 5. Stability of evaluation performance according to the number of benchmark samples[1]
In contrast, ATAD defines a benchmark not as a fixed set of data, but as a protocol through which the Teacher-Orchestrator-Student interact. Currently, when the Student solves a problem, the Teacher generates a more difficult problem, and the Orchestrator validates that the problem remains clear and fair. Therefore, even if a more powerful model emerges to take on the role of the Student, the difficulty of the evaluation can be raised again to match that model’s capabilities.
This enables Model-Benchmark Co-evolution, in which the benchmark and the model do not exist independently of one another, but rather the benchmark evolves alongside the model as it advances. The paper itself emphasizes that shifting the focus from traditional static dataset-based evaluation to dynamic protocol-based evaluation is a core research direction of ATAD.
Furthermore, the fact that problems are generated dynamically does not mean that evaluation results vary significantly each time. In experiments where benchmarks were repeatedly generated and evaluated using the same agent configuration, the model’s average performance remained relatively stable even as the number of samples increased. This demonstrates that ATAD can adapt to the model’s capabilities while maintaining a certain level of consistency and stability as an evaluation protocol.
Figure 5. Stability of evaluation performance according to the number of benchmark samples[1]
Ultimately, the goal of ATAD is not simply to “create yet another more challenging dataset,” but to ensure that evaluation methods evolve alongside the AI being evaluated.
Future Research Directions: Toward a Self-Evolving AI Evaluation System
This study validated the potential of ATAD with a focus on Text Anomaly Detection. Unlike mathematics or coding, Text Anomaly Detection makes it difficult to validate answers using clear formulas or execution results, while simultaneously requiring various forms of reasoning, such as context, logic, referential relationships, and style. Therefore, it served as a challenging environment for validating whether the difficulty level could be increased while maintaining the clarity of the problems. Due to these characteristics, the paper also utilized Text Anomaly Detection as a representative stress test for ATAD.
However, the Teacher-Orchestrator-Student structure of ATAD itself is not dependent on any specific task. In the future, this framework can be extended to a wider range of domains, such as mathematical reasoning, code validation, tool use, and multimodal reasoning. Additionally, we can consider providing the Teacher with external documents or knowledge to generate domain-specific problems, or integrating Retrieval-Augmented Generation (RAG) or a separate verifier with the Orchestrator to validate factual accuracy and logical validity more rigorously.
Furthermore, while humans currently design the roles and workflows of the Teacher, Student, and Orchestrator, in the future, Meta-Agents could be used to automatically identify better Teachers and Orchestrators or to optimize the roles and interactions among Agents. This could lead to the development of a system in which not only the evaluation problems but also the Agents that create them and the evaluation protocols themselves evolve together. The paper also proposed follow-up research directions, such as analyzing the agents’ contributions from a game-theoretic perspective or automatically exploring more effective agent workflows through Meta-Agents.
Ultimately, LG AI Research aims to move beyond an evaluation approach that requires rebuilding new benchmarks from scratch every time a model evolves, and instead expand its research toward a dynamic evaluation system in which the difficulty of the problems and the evaluation methods continuously adapt based on the AI’s capabilities. The goal is to develop evaluation methods that go beyond simply comparing a model’s final score, allowing for continuous exploration of the model’s reasoning capabilities and the specific points at which it fails.
At ICLR 2026, we presented our ATAD research and shared our view that evaluation methods must evolve alongside the rapid advancements in large language models (LLMs). Moving forward, we plan to continue leveraging interactions between agents to measure a model’s capabilities and limitations more precisely, and to develop evaluation methods that remain valid even as new AI technologies emerge.
- 참고
- [1] Seungdong Yoa, Sanghyu Yoon, Suhee Yoon, Dongmin Kim, Ye Seul Sim, Junhyun Lee*, Woohyung Lim*. From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning. In International Conference on Learning Representations (ICLR), 2026.
[2] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR), 2021.
[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman. Training Verifiers to Solve Math Word Problems.