[AAAI 2026] RL-Studio: A System for Multi-Phase Reinforcement Learning Experimentation

Introduction

The AAAI(Association for the Advancement of Artificial Intelligence) is one of the most prestigious international conferences in the field of artificial intelligence, and every year, AI researchers from around the world gather to share the latest research findings and discuss the future direction of technology.

The most actively presented research topic at AAAI 2026 was the field of “Physical AI.” Interest in AI systems operating within the physical world has significantly increased, as evidenced by the addition of the “Embodied AI” keyword to the Intelligent Robotics category. Within this trend, research on reinforcement learning (RL), a core technology in robot learning, also occupies an important position.

In accordance with these changes, LG AI Research's Physical Intelligence (PI) Lab unveiled RL-Studio, a reinforcement learning experimental platform, through the Demonstration Program at AAAI 2026.

What is reinforcement learning?

Reinforcement learning (RL) is a learning method where an AI agent finds optimal actions through trial and error with its environment. Just as a child learns to ride a bicycle by falling and getting up multiple times to master balance, an RL agent also interacts directly with its environment and gradually learns better strategies (policies) based on reward signals. Due to these characteristics, reinforcement learning plays a pivotal role in various fields such as game AI, robot control, and autonomous driving.

Traditional RL involves agents directly interacting with the environment to collect and learn data in real time. However, this approach requires numerous environmental interactions to gather sufficient data and faces limitations in environments where trial-and-error costs are high, such as with actual robots or industrial systems.

Offline reinforcement learning (Offline RL) emerged to overcome these limitations. Offline RL is a method that learns solely using a pre-collected, fixed dataset (offline dataset), without requiring additional interactions with the environment. With the emergence of offline RL, the traditional environment interaction-based learning approach came to be referred to as online reinforcement learning (Online RL) to distinguish them.

Recently, the technology has evolved into Offline-to-Online RL, which combines the strengths of both approaches. This hybrid method first pre-trains using Offline RL to learn a stable initial policy, then transitions to Online RL for further training in the actual environment. 

Figure 1. An overview of Online, Offline, Offline-to-Online RL. Offline RL learns from pre-collected datasets, while online RL learns by collecting data through interactions with the environment. Offline-to-Online RL involves a phase transition where an offline pre-trained policy is transferred to the online phase for continuous optimization

However, in real-world industrial settings where complex variables exist, more sophisticated and intricate processes are often required that cannot be adequately addressed by these three approaches alone. This post introduces LG AI Research PI Lab's Multi-Phase RL framework, presented in the AAAI 2026 Demonstration Program, and the supporting platform RL-Studio.

1. The beginning of research: Limitations of existing RL paradigms

When conducting experiments in real-world industrial settings, RL researchers face numerous practical challenges. Problems become particularly complex when connecting different learning approaches and combining them in complicated ways.

The RL paradigm has evolved through Online RL, Offline RL, and the combined Offline-to-Online RL. While Offline-to-Online RL offers a practical approach by pre-training on existing data and then fine-tuning in the real environment, its learning process is confined to a fixed two-step sequence: “Offline → Online.”

At this, LG AI Research posed a fundamental question that broke the mold. "What lies beyond offline-to-online RL?"


● Why multi-phase RL?

Offline-to-Online RL is actually just one example of the broad paradigm we may encounter. From the perspective of transfer learning or sim-to-real adaptation, transitions during the learning process can occur in far more diverse forms.

- Transition of learning styles: Flexible transitions between Offline ↔ Online
- Transition of algorithms: For example, training with TD3 and then transitioning to SAC and continuing training
- Transition of the learning environment: Sim-to-real scenarios that apply policies learned in simulation to real environments

LG AI Research demonstrates through this paper how much these transitions by step contribute to actual performance improvement via two representative experiments.

1. Fine-Tuning scenario: This involves fine-tuning a pre-trained policy using offline RL (TD3-BC^[1]) with online RL (TD3^[2]). Thanks to the pre-trained policy, higher sample efficiency is achieved compared to pure offline RL or starting from scratch with online RL (Figure 2)

2. Adaptation scenario: This case involves learning using Online RL (TD3) in a simulation environment (1.0G gravity) and then transitioning to another algorithm (SAC^[3]) to adapt in the actual environment (1.2G gravity). It can be observed that convergence occurs much faster than when learning from scratch. (Figure 3)

Figure 2. TD3-BC (Offline learning with pre-collected data) -> TD3 (Online learning by interacting with the environment)

 

Figure 3: TD3 (online learning in simulation) -> SAC (online learning in actual environment)

In environments like this, where flexible combinations of multiple learning phases are required, existing frameworks have been limited to supporting only predefined learning transitions (e.g., offline to online) or being restricted to a single algorithm. These scenarios raise important research questions. 1)When should we transition phases? 2)Which combination of phases is optimal for a given task? 3)How can we predict the outcome of a transition? To systematically address these questions, LG AI Research has developed a multi-phase RL framework and the supporting platform RL-Studio.

2. Multi-Phase RL and RL-Studio 

Multi-Phase RL Framework

LG AI Research proposes a multi-phase RL framework as a method to overcome existing limitations. Multi-phase RL extends beyond the “two-phase” structure of Offline-to-Online RL, enabling flexible transitions across arbitrary sequence of learning phases. Each phase can independently configure its learning method (Online, Offline, Offline-to-Online), algorithm, and learning environment, allowing for the flexible construction of complex learning scenarios in industrial settings, as mentioned earlier.

RL-Studio: An experimental platform for multi-phase RL

RL-Studio is a platform designed to enable easy and efficient execution of multi-phase RL experiments. Previously, configuring a new phase of an experiment required significant time spent on repetitive setup tasks such as algorithm configuration, loading the learned model (checkpoint) from the previous phase, and connecting the dataset. RL-Studio reduces this experimental setup overhead, allowing researchers to focus on the core research questions of multi-phase RL.

The main features supported by RL-Studio are as follows.

1. Support for various environments: OpenAI Gym^[7] compatible environments such as MuJoCo^[4], AntMaze^[5], Adroit^[6], etc.
2. Broad range of algorithms: Various algorithms for Online RL, Offline RL, and Offline-to-Online RL
3. Flexible dataset utilization: D4RL dataset^[8] and compatible custom datasets
4. Easy expansion: Possible to add custom algorithms or environments as needed

Notably, this RL-Studio also incorporates the Offline-to-Online RL algorithms PARS^[9] and OPT^[10], which LG AI Research presented at ICML 2025, enabling users to immediately leverage the latest research findings in their experiments.

Demo video 

The videos below demonstrate the process of setting up a multi-phase RL experiment and verifying the results using RL-Studio.

 

Video 1: The first phase of the experiment setup. This shows the process of selecting the environment and dataset and starting training with the offline RL algorithm TD3-BC.

Video 2: The second phase of the experiment setup. This shows the process of loading the checkpoint learned in the first phase and then transitioning to TD3, an online RL algorithm, to continue learning.

Video 3: Comparison of graphs from multiple experiments. As online learning measures performance by the number of interactions with the environment (interaction steps), and offline learning measures performance by the number of learning iterations (gradient steps), RL-Studio automatically separates graphs by learning paradigm to facilitate fair comparison.

Like so, using RL-Studio allows you to intuitively configure multi-phase RL experiments and easily perform model transitions between phases and compare performance.

3. Related research at AAAI 2026 

In offline-to-online RL, policy optimization consists of two main objectives. One is policy improvement, which maximizes the state-action value function (Q-value) to refine the policy, and the other is policy constraint, which restricts the policy from deviating significantly from the existing data distribution. Performing policy improvement too aggressively early in online fine-tuning can amplify distribution shift, leading to unstable performance, while conversely, applying policy constraints too strongly can result in conservative learning, limiting performance gains. Existing offline-to-online RL algorithms have focused on balancing these two objectives, but most have the limitation of applying the same balance to all samples.

SPA (State Proficiency-Based Adaptive Fine-Tuning)^[11], presented at AAAI 2026, addresses this issue at the sample level. SPA's key observation highlights that the effectiveness of offline pre-training varies significantly across states due to the gap between the dataset's limitations and the policy's generalization ability. Policies learn effectively in states well-represented in the dataset with high-quality actions, but learning is insufficient in states outside the data distribution or where only suboptimal actions exist. SPA defines this variation in learning effectiveness as the concept of state proficiency and proposes an optimal learning strategy based on it.

Evaluation of SPA's state proficiency 

SPA uses an action comparison-based classification method to evaluate state proficiency. The fundamental idea is to compare the quality of actions taken by the policy currently being trained against a reference policy. Specifically, for each state s, it compares the Q-value of the current policy π, Q(s, π(s)), against the Q-value of the reference policy, Q(s, π_ref(s)). If the current policy's Q-value is higher than the reference policy's, the state is classified as high proficiency; otherwise, it is classified as low proficiency.

To reflect the differing characteristics of offline and online data, the SPA introduces dual thresholds. For offline data, it uses a fixed offline policy (π_off) as the reference policy and sets a relatively low threshold, while for online data, it uses the optimal policy (π_opt) learned during the learning process as the reference policy and sets a relatively high threshold. This achieves the effect of learning conservatively with offline data and aggressively with online data.

Furthermore, due to the issue of Q-value overestimation potentially leading to incorrect proficiency classification, SPA performs classification correction for state proficiency using a separate state value function (V-function). Among samples classified as high proficiency, if Q(s, π(s)) is lower than V(s), it is judged to be misclassified due to overestimation and reclassified as low proficiency.

Adaptive learning strategies in SPA

Once proficiency-based evaluation is complete, SPA applies different learning strategies to each sample based on the classification results. For samples with low proficiency, it increases the weight of policy improvement to actively enhance performance, and for samples with high proficiency, it increases the weight of policy constraint to stably maintain the already learned performance. Through this dynamic balancing mechanism, it aggressively explores when learning is insufficient, then automatically transitions to conservative learning as proficiency increases, and conversely, when performance degrades, it rapidly recovers by reverting to active learning.

As a result, SPA achieved performance improvements of 64.6% and 207.9% over existing methods on the Locomotion and Maze2D tasks of the D4RL benchmark, setting a new state-of-the-art (SOTA).

The Offline-to-Online RL field is actively developing, and RL-Studio provides an environment where these latest Online, Offline, and Offline-to-Online RL algorithms can be easily experimented with and compared.

4. LG AI Research's future plans  

LG AI Research unveiled RL-Studio through the AAAI 2026 Demonstration Program, engaging with researchers worldwide and collecting valuable feedback. Based on this, we plan to enhance the platform's completeness and further advance multi-phase RL research through collaboration with the reinforcement learning community.

Long-term, the goal is to apply RL-Studio to real robots or industrial systems, evolving it into a closed-loop system where agents learn in simulation, deploy to real-world environments, and then utilize operational data back into the learning process. LG AI Research plans to continue pioneering innovative research in reinforcement learning and lead the development of AI technologies applicable in real industrial settings.
 

▶RL-Studio: Learn more: A System for Multi-Phase Reinforcement Learning Experimentation Paper