[AAMAS 2024] Multi-Agent Reinforcement Learning for Real-World Application (Part 2)

In Part 1, we explored examples and studies that apply multi-agent reinforcement learning (MARL) to real-world problems. While AI models have demonstrated potential in solving real-world problems, there are still many factors to consider when implementing them in real-world scenarios. Can we make agents learn more efficiently to optimize costs? What if it’s difficult to develop a simulator that can interact with agents? How can we utilize the vast amounts of data being accumulated at industrial sites? This post will take a look at three papers presented at AAMAS 2024 that address these questions..

Multi-Agent Reinforcement Learning for Real-World Application Part 1 Data Intelligence Lab, Deunsol Yoon (Link) Part 2 Data Intelligence Lab Sunghoon Hong (this post)

 

1. Agent-Oriented Centralized Critic for Asynchronous Multi-Agent Reinforcement Learning^[1]

In real-world problems, such as the NCC scheduling optimization problem introduced in Part 1, the duration or start time of an action may be different for each agent. These problems are approached with Asynchronous MARL, but existing studies have mainly focused on how to handle asynchronously collected data and proposed methods. In this context, LG AI Research proposes a new architecture to learn more efficiently through this study.

Asynchronous MARL

Many MARL problems assume that each agent is only partially observable and acts based on the partial observation, and learns a value function over the agent’s observations or history instead of the actual state. This training structure is called Centralized Training Decentralized Execution (CTDE) and typically involves training two models: a Decentralized Actor that determines the behavior and a Centralized Critic that evaluates the value. Similarly, the same structure can be leveraged in Asynchronous MARL.

This study specifically addresses the Macro-action Decentralized Partially Observable Markov Decision Process (MacDec-POMDP)^[2], which is an Asynchronous MARL problem that assumes that each agent performs a macro-action of different lengths and can observe only when the macro-action starts or ends. Unlike in a typical MARL problem, where each agent’s latest observations are recorded at each timestep, making it easy to construct a history, in this case, each agent’s observations may not be up-to-date, so it is needed to think about how to construct a history to train the value function.

Image 1. Two conceptual diagrams of Experience Replay Trajectories^[1]

Existing studies suggest that histories can be constructed in two ways in the Asynchronous MARL problem based on observations and action histories (Experience Replay Trajectories, ERTs) of all agents^[3]. One is Mac-CERTs (Macro-action Concurrent ERTs), where each agent independently constructs its own history based on the timing of its macro-actions, and the other is Mac-JERTs (Macro-action Joint ERTs), where each new observation is included in the history whenever any agent makes a new observation. Consequently, the previous studies propose to use Mac-CERTs for decentralized actors, as they are independently constructed for each agent, while using Mac-JERTs for learning a value function, i.e. a centralized critic, based on the comprehensive history of all agents.

Agent-Oriented Centralized Critic

If we were to directly use Mac-JERTs as input to critic, we would also have to provide the history of the agents at the timestep even when they do not have the latest observations, so in previous method, the last observation was conimaged as input instead. However, this method may result in outdated observations being duplicated as inputs, potentially failing to appropriately represent the history.

This study proposes a new architecture called Agent-Oriented Centralized Critic (AOCC), which consists of an agent-oriented history encoder and an aggregation module. The agent-oriented history encoder encodes agent-specific histories consisting of Mac-CERTs, and the aggregation module aggregates the history encodings of each agent. Positional encoding is added for each timestep to the input, as simply using a history consisting of Mac-CERTs would confuse the timing of observations between agents. The resulting AOCC avoids the centralized critic of existing studies, which contains duplicate observations, and better organizes history to learn value functions and action policies as follows.

Image 2. (Left) Centralized Critic based on joint history (Right) Agent-Oriented Centralized Critic conceptual diagram^[1]

Experimentally, in a reinforcement learning environment based on the popular video game "Overcooked," AOCC showed better performance in terms of learning speed and final performance compared to previously proposed methods such as NIACC and IAICC. This indicates that improvements in the learning architecture for Asynchronous MARL can impact learning performance.

Image 3. Comparison of experiment results in asynchronous MARL^[1]

So far, this post has introduced studies on improving the efficiency of Asynchronous MARL methods under realistic conditions. This study is applicable to cases in which a simulated environment is developed and agents learn action policies by directly interacting with the environment without any prior information. However, in reality, there may not be a simulator for reinforcement learning for the problem to solve. How can we solve this problem? Next, this post will look at papers that address these concerns.

2. A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem^[4]

Industries sites are constantly accumulating data of experts’ actions in the real world, not in simulations, so why not use data to train agents rather than simulations? Training agents through interactions in a simulator is known as Online Reinforcement Learning (Online RL), whereas training using only pre-collected data without simulations is referred to as Offline Reinforcement Learning (Offline RL). Offline RL can be a good alternative in situations where simulators are difficult to obtain. This study proposes to extend offline RL for multi-agent problems to train agents to a certain level using data alone.

Offline Reinforcement Learning

Offline RL is a method for learning an agent’s action policy solely from past accumulated data without any additional interaction between the agent and the environment, in other words, no additional data collection. The most important difference here from typical online RL is that it is not possible for the agent to try out its action policy in the environment and receive new feedback.

As typical RL algorithms are designed based on the trial-and-error of experiencing new actions and states, simply applying them to offline RL problems can lead to poor learning due to overestimation of states and actions that have not been experienced. To compensate for these limitations, many offline RL algorithms have proposed to conservatively learn action policy or value functions on states or actions that have never been seen in the data. Examples include Conservative Q-Learning[5], which adds an objective function that values actions in the data highly and new actions less highly, and TD3+BC[6], which adds an objective function that mimics actions in the data.

Offline Multi-Agent Reinforcement Learning

While existing offline RL algorithms have shown that they can compensate for the overestimation problem when training a single agent, they are limited in their ability to simply extend to offline MARL. This study points to two main problems: strategy agreement and strategy fine-tuning. Unlike single-agent problems, multi-agent problems require multiple agents to agree on a common team strategy for cooperation (strategy agreement), while at the same time adjusting their own strategies according to the agreed team strategy (strategy fine-tuning). In online MARL, these problems can be solved through numerous rounds of trial and error in the environment, but in offline MARL, it is difficult to agree on the correct team strategy and to adjust each individual strategy accordingly, because the different strategies are fragmented and have to be learned from limited recorded data.

Image 4. Comparison of online and offline MARL^[4]

Model-Based Offline Multi-Agent PPO (MOMA-PPO)

This study points out that this lack of additional interaction is one of the main weaknesses of offline MARL. To address this, it proposes an alternative method to learn an environment model and train the agent based on it, which is called Model-based Reinforcement Learning (Model-based RL). In this context, an environment model is a model that predicts the next state and reward for a given state and action. With this model, it becomes possible to interact with a virtual environment instead of interacting with a real environment. As the virtual environment model is trained with deep learning, it cannot provide the exactly same results as the real environment, but it can still be a good alternative for additional interaction.

By training an environment model from data and putting in place some techniques to compensate for the imperfections of this model, it is possible to apply typical online RL algorithms. In this study, N models are trained independently and the variance between them is used to estimate the epistemic uncertainty of the models, and this is used to supplement the predicted reward by adding a penalty proportional to the epistemic uncertainty, or by preventing the model from transitioning to states with epistemic uncertainty higher than a certain threshold. Finally, this paper proposes a method for learning agents on a virtual environment model, namely MOMA-PPO, through the MAPPO algorithm, which extends PPO, an RL algorithm designed for single agents, to multi-agents.

Image 5. Comparison of performance of offline MARL experiments^[4]

The proposed method in this study, MOMA-PPO, performs much better than existing offline MARL methods in various experiments such as robot control. This demonstrates that learning environment models and compensating for their imperfections, along with stable PPO algorithm-based training, are effective in offline MARL. In conclusion, even if it is difficult to obtain a simulation environment in a multi-agent setting, it is expected that it is possible to train agents to a certain level if there is accumulated data from the real world.

3. Improving Offline-to-Online Reinforcement Learning with Q-Ensembles^[7]

An important consideration when using AI models in the real world is learning efficiency. Improving learning efficiency is an important issue, especially with the recent rise of methods like Large Language Models (LLMs) that are astronomically expensive to train. Previous studies have shown that it is possible to train an agent to a certain level using historical data without interacting with the environment. What if pre-trained agents could learn further by interacting with the environment? This paper discusses Offline-to-Online RL, a methodology for increasing the efficiency of learning.

Offline-to-Online Reinforcement Learning

Offline RL is a methodology for training a certain level of agents with limited data, and Online RL is a methodology for training the most optimally performing agents by interacting with the environment. Offline-to-Online RL is a combination of the two, where an agent learns to a certain level with pre-collected data (offline pre-training) and then quickly increases its final performance through additional interactions with the environment (online fine-tuning). While it may seem like learning would work well if each method was properly connected, the distribution shift causes a number of problems. This means that in reality, the action policy used to collect the data may be different from the state and action that is newly experienced in the real world. This can lead to drastic drop in performance at the beginning or slower training speeds in online fine-tuning.

Ensemble-Based Offline-to-Online Reinforcement Learning

This study experimentally demonstrates that the problem caused by distribution shift can be significantly mitigated by using an ensemble-based value function. In general RL, one or two value functions, known as Q-functions, are typically learned to estimate the value of a state or action. However, using N value functions instead, properly combined, can lead to more stable learning of value functions and significantly reduce volatility. This approach helps prevent performance degradation even in the presence of distribution shifts and accelerates faster learning.

Image 6. Comparison of performance with and without ensemble application^[7]

And experimentally, the MinPair approach seems most suitable among various ensemble techniques. This method involves selecting any pair from N trained Q-functions and using the minimum value among them. Also, as only limited data was experienced in the offline pre-training phase, exploration needs to be emphasized more in the online learning phase, and among the various exploration strategies, SUNRISE[8] seems to be the most effective. SUNRISE is an exploration strategy in ensemble-based RL algorithms that uses the standard deviation of multiple Q-functions to stabilize the learning of the Q-function and allow itan agent to explore states and actions it has not yet experienced.

1. Updating the Q-function proportional to the confidence weight:
   $L_{W}Q\left({\theta }_i\right)=w(s_{t+1},a_{t+1})\left(Q_{{\theta }_i}\right(s_t,a_t)-r_t-\gamma \overline{V}(s_{t+1}){)}^2$
   $,where a_{t+1}~\pi (a_t│s_t),w(s,a)=\sigma (-{\overline{Q}}_{std}(s,a\left)T\right)+0.5,T>0,\sigma is sigmoid function$

2. An exploration strategy that searches for actions with high uncertainty
   $a_t=\underset{a}{max}\left\{Q_{mean}\right(s_t,a)+\lambda Q_{std}(s_t,a)\}$
   
   

Image 7. (Left) Comparison of ensemble techniques (Right) Comparison of exploration strategies^[7]

By integrating these methods, this study proposed a method called E2O, which performed excellently on several benchmarks. Of course, it showed very fast training speed and final performance compared to online RL algorithms that learn from scratch without prior training using data. It can be seen that it is possible to train an agent very efficiently by utilizing accumulated data and additional training.

Image 8. Offline-to-Online performance comparison^[7]

Conclusion

So far, this post has covered the studies presented at AAMAS 2024, including Asynchronous MARL, Offline MARL, and Offline-to-Online RL research that is needed to apply RL to real-world problems. With the recent increase in interest in RL, studies on MARL have been gaining traction. To address real-world problems, it is important to consider a number of realistic conditions, such as utilizing accumulated data and asynchronization between multiple agents. There are still many things to improve in each research field, but if we can advance each technology and further draw out synergies between them, we can expect to become more efficient in the actual industrial field. LG AI Research will continue to listen to voices from the real industrial field and lead the research of AI technology for a better life.

 

▶Agent-Oriented Centralized Critic for Asynchronous Multi-Agent Reinforcement Learning

▶Naphtha Cracking Center Scheduling Optimization using Multi-Agent Reinforcement Learning