Reinforcement learning is an AI technology in which an agent interacts with an environment to learn how to make better decisions. It can be applied not only to robot control, but also to problems requiring continuous decision-making, such as factory operations, logistics, and production planning. In this context, performance is not determined solely by the RL algorithm and the reward function. The observation space, which determines what information the agent receives, and the action space, which determines the scope and units within which decisions are made, are also critical factors, serving as the agent’s “eyes and hands,” respectively. If observations and actions are not designed to suit the characteristics of the task, the agent may require more time and data to learn an effective policy.
The 43rd International Conference on Machine Learning (ICML 2026), held in Seoul this July, is a leading international conference covering a wide range of topics, from fundamental research in machine learning to applied research in robotics, life sciences, and data science. At this conference, LG AI Research presented a study titled “Designing Observation and Action Models for Efficient Reinforcement Learning with LLMs”[1], in which large language models (LLMs) directly design the observation and action spaces. It was a meaningful opportunity to share questions arising from real-world industrial settings with the global research community and discuss the potential for automating environment design with various researchers.
Image 1. The ICML 2026 Poster Session, where the core ideas and experimental results of LOAM were presented and discussed with attending researchers
1. A question that arose in the industrial setting: What should AI observe in order to make decisions?
1.1 A question originating from the NCC project
The starting point for this study was the “NCC Operations Optimization AI Scheduler Development Project” project [2,3,4], in which we have been participating since 2022. This project was a joint initiative between LG AI Research and LG Chem, aimed at developing a reinforcement learning-based AI scheduler to optimize the complex production scheduling at LG Chem’s Daesan NCC plant and thereby improve profitability. An NCC (Naphtha Cracking Center) is a complex process that thermally cracks naphtha to produce basic petrochemical feedstocks such as ethylene and propylene. At the time, our team began the reinforcement learning-based production scheduling optimization project with virtually no prior knowledge of NCC. We conducted numerous interviews with on-site personnel to understand how the plant operates, what information field experts rely on to make decisions, and what constraints apply under normal and exceptional circumstances. Looking back, we spent more time understanding the on-site operations and translating experts’ judgments into an observation·action·reward framework than we did implementing and training the reinforcement learning algorithm.
1.2 Observation and action design, more difficult than rewardThrough this experience, we came to realize that applying reinforcement learning to the real world requires careful design not only of the reward function but also of the observation space and action space. In the NCC project, the optimization goals and evaluation metrics were relatively clear, so the process of defining the reward function was fairly intuitive. On the other hand, it was difficult to determine which information among the numerous operational variables should be provided as input to the policy, or how to structure the action space so that the policy could make decisions within a specific range and unit. This design required the experience of field experts and tacit operational knowledge that is difficult to express in writing. It also raised the question of whether reinforcement learning experts and field experts would have to repeat this process from scratch every time a new industrial task was assigned.
Subsequently, with the emergence of research, such as EUREKA[5] and Text2Reward[6], that utilizes LLMs to design reward functions, we confirmed the potential to use LLMs’ semantic reasoning capabilities to design other components of reinforcement learning environments as well. Compared to reward design, research on observation and action space design has been relatively scarce, and this was also the area where the NCC project actually encountered the greatest difficulties. This awareness of the issue led to the LOAM research, which utilizes LLMs to design observation and action models.2. Designing observations and actions with LLMs: LOAM2.1 Environmental design beyond reward
A reinforcement learning environment consists primarily of observation, action, and reward. While recent LLM-based research has focused on translating task objectives expressed in natural language into reward model code, the observation model and action model are still often designed manually by humans for each task. However, the decision of which raw sensor data and state variables to present to the policy, and how to map the policy’s output to actual control variables, directly impacts learning speed and final performance.
Image 2. Redesigning the reinforcement learning environment through observation and action models
2.2 LOAM: Observation and action models created by LLMs
Instead of directly feeding the information provided by the environment into the policy, LOAM is designed to distinguish between the raw environment space and the agent space actually used by the reinforcement learning agent. The raw environment space consists of raw observations, which are all state information obtainable from the simulator, and raw actions, which are all control commands that can be sent to the robot’s actuators. When LOAM provides the LLM with the task’s goal and termination conditions, the robot’s structure, and the meanings of each variable, the LLM uses this information to generate an observation model and an action model, each in the form of a Python function. The observation model converts raw observations into agent observations, which the policy uses as input, while the action model converts the agent actions output by the policy into raw actions that the environment can execute. Through this process, LOAM reconfigures the interface between the raw environment space and the agent space to suit the specific task.
Image 3. LOAM that generates observation and action functions from task, robot, and state descriptions
Below is a condensed example of the observation model generated by LLM in the HumanoidBench run task.
Image 4. Example of an abbreviated observation model for the HumanoidBench running task generated by LLM
This function does not simply pass on information from the simulator; instead, it constructs features necessary for the task, such as converting speed and direction to the robot’s frame of reference or calculating the difference between the current speed and the target speed. While the observation model transforms information from the environment into a format that the policy can use, the action model maps the structured action output by the policy into a raw action that the actual actuator can execute. Since both functions act as wrappers around the existing environment, they can be applied without modifying the reinforcement learning algorithm itself.
This observation·action design led to actual improvements in learning efficiency. In the stand·walk·run tasks, the basic locomotion tasks on HumanoidBench, LOAM reached a high return more than three times faster than the human-designed FastTD3 baseline environment. This demonstrates that the model generated by the LLM does more than simply produce executable code; it can also reconfigure the environment into a form that is conducive to the policy learning the task.
Image 5. HumanoidBench stand·walk·run task training results.
The prompt is also designed to be modular so that it can be easily adapted to new tasks and domains. When the environment changes, you only need to provide updated descriptions of the task’s objectives, the robot’s structure, and the available observations and actions; the rest of the prompt can be reused as-is. Most of this information can be obtained from environment documentation or APIs. Therefore, rather than having reinforcement learning experts and domain experts manually design the observation and action spaces from scratch for each task, the LLM can generate an initial design proposal simply by organizing the information already provided.
2.3 LOAM-Race: Finding a good design on a limited budget
Even when fed the same prompt, an LLM generates slightly different results each time. Consequently, the observation model and action model created by LOAM also vary in composition each time, leading to differences in the performance of the reinforcement learning policy trained based on them. Previous studies have primarily used an approach that generates multiple candidates and evaluates them iteratively to reduce variations in the quality of LLM outputs. For example, EUREKA generates multiple reward function candidates, trains a policy using each reward function, and evaluates its performance. It then feeds the training statistics of the best candidate and policy back into the LLM to generate new candidates, repeating this process multiple times to improve the reward function.
The problem is that, in order to assess the quality of each candidate, the reinforcement learning policy must be trained until it has sufficiently converged. Repeating the process of generating and evaluating multiple candidates to find a better design requires a correspondingly large number of simulations and a significant amount of training time. Ultimately, as reinforcement learning must be performed multiple times even before the final candidate is selected, the entire process incurs considerable time and computational costs.
Image 6. LOAM-Race, which dynamically allocates learning resources by predicting the potential of candidates
To address this issue, we propose LOAM-Race, which pits multiple design candidates against each other within a single training process. First, we briefly train the observation and action model candidates generated by the LLM, and then predict the performance each candidate is likely to achieve in the future based on its current performance and learning curve. If we select only the candidate with the highest current performance, the initial training speed will be slow, but we may miss out on candidates that could deliver better performance later on. Therefore, LOAM-Race uses Bayesian Ridge Regression to consider not only the expected performance but also the uncertainty of the prediction.
Next, the next training interval is allocated to the most promising candidate, and the potential of all candidates is reevaluated based on the newly obtained results. By repeating this process, we can allocate minimal resources to candidates with low performance and little room for improvement, while focusing training on promising candidates or those whose potential has not yet been fully confirmed. In other words, it is possible to explore and train a good observation·action model within the training budget of a typical single policy, without having to repeatedly perform separate, large-scale training sessions for candidate selection.
Image 7. LOAM-Race, where multiple observation and action design candidates compete during the learning process
2.4 Key Results
Experiment Environment
We validated LOAM’s performance using two robot control benchmarks with distinct characteristics. HumanoidBench[7] is a simulation environment in which humanoid robots perform tasks ranging from locomotion tasks, such as standing, walking, and running, to complex manipulation tasks, such as pushing objects, manipulating cubes, and carrying boxes. A total of 12 tasks were evaluated, divided into three categories: locomotion, static manipulation, and dynamic manipulation. Isaac Lab used three manipulation tasks: “Lift-Cube” and “Open-Drawer” with the Franka robot arm, and “Repose-Cube,” in which the Allegro Hand changes the orientation of a cube within its grasp. We applied the same reinforcement learning algorithm, FastTD3[8], to all experiments.
HumanoidBench Results
LOAM achieved higher performance more than three times faster than the human-designed FastTD3 baseline on the basic Locomotion tasks, stand, walk, and run, in HumanoidBench. Not only in locomotion but also in most tasks, including static and dynamic manipulation, it demonstrated faster learning speeds or higher final performance than FastTD3 and LESR[9]. In particular, unlike LESR, which adds features generated by an LLM to existing human-designed observations, LOAM differs in that it reconstructs the information necessary for the task from raw observations and simultaneously designs an efficient action space. Since LOAM-Race allocates the training budget across multiple candidates, its initial learning rate may be slightly slower than that of the single LOAM method. However, for tasks such as Cube and Insert_small, where the quality of LLM-generated designs varies significantly, LOAM-Race achieved higher final performance more reliably by selecting promising candidates through the racing process. On the other hand, there were tasks such as Package where all methods struggled during training, indicating that complex long-term manipulation problems remain an area requiring further research.
Image 10. HumanoidBench experiment results
Isaac Lab Results
Isaac Lab evaluated whether LOAM’s observation·action design could generalize to different simulation environments and robot embodiments. In the Lift-Cube and Open-Drawer tasks using the Franka robot arm, LOAM learned faster than the baseline, and in the Repose-Cube task using the Allegro Hand with over 1,100-dimensional raw observations, LOAM-Race demonstrated the highest final performance. This demonstrates the importance of jointly designing observations, which determine “what to observe”, and actions, which determine “how to move”, in reinforcement learning.
Image 11. Isaac Lab experiment results
In addition, the paper includes further experiments and analyses, such as the individual effects of the observation model and action model, differences based on LOAM-Race’s candidate selection method and the number of candidates, and performance across various LLM backbones. For those interested in the specific operations and results of LOAM and LOAM-Race, please refer to the paper.
3. Future Plans
In the future, we plan to expand LOAM to various robot types and tasks, including LG’s home robots and robots used in manufacturing facilities. As each robot has a different configuration of sensors and actuators and performs different tasks, we plan to use LOAM to efficiently design observation·action Models suited to each specific environment. To this end, we plan to organize robot specifications, environment APIs, and task-related technical documentation into a format that LLMs can understand, and strengthen procedures to automatically verify the feasibility, physical validity, and safety of the generated observation·action models through simulation. Ultimately, we aim to actively utilize LOAM in real-world robot deployments by establishing a collaborative system where robot and field experts describe the task’s objectives, constraints, and exception scenarios, allowing the LLM to construct an initial model, while reinforcement learning experts focus on policy learning and validation.
Deunsol Yoon
2026.10.26
