[ICLR 2021] Part 1: Reinforcement Learning for Real-World Problems

International Conference on Learning Representations (ICLR) 2021 is the 9th global conference on artificial intelligence and learning that primarily deals with the representation, learning, and optimization of deep learning models and their applications. Among a variety of academic papers available at ICLR 2021, which was held online from May 3 to 7, two researchers of LG AI Research who participated in the event would like to introduce to you two papers they have found particularly impressive. - Part 1: Reinforcement Learning for Real-World Problems - Researcher Youngjoon Park - Part 2: Generative Model Trends at ICLR 2021 - Researcher Sanghyu Yoon

ICLR 2021 was held online from May 3 to 7. Because it was an online conference, I could watch the presentations at any time I wanted without effort to get over jet lag. Most of the presentations I watched are reinforcement learning studies in which I have a particular interest. To identify the topics discussed in the conference, I visualized the keywords of the papers accepted at this conference on word cloud. Papers were published on a wide variety of topics throughout reinforcement learning. Comparing to ICLR 2020 held the previous year, it was clear that studies on robot control such as robotics and goal-reaching tasks have relatively increased. From this fact, we can think that recent studies on reinforcement learning do not remain merely focused on computer games but are also consider problems applicable to industries.

Word Cloud for keywords on reinforcement learning at ICLR 2021

Reflecting these recent trends, I have selected and organized the research published under the topic of ‘Reinforcement Learning for Real-World Problems.’ Although reinforcement learning is a useful tool for solving sequential decision-making problems, there are many challenges in applying it to real problems outside of simulated environments such as Atari games, like many machine learning techniques suffer. At ICLR 2021, many studies were published to address this difficulty. Among them, I selected papers that focused on the two study topics of 1) Unsupervised Reinforcement Learning aims to address the difficulty due to sparse rewards, and 2) Reinforcement Learning with Self-Supervision to achieve learning efficiency and robustness when states are observed as a form of visual input such as images.

1. Unsupervised Reinforcement Learning ■ Mutual Information State Intrinsic Control (Zhao et al., 2021)[5] ■ SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (Berseth et al., 2021)[2] 2. Reinforcement Learning with Self-Supervision ■ Learning Invariant Representations for Reinforcement Learning without Reconstruction (Zhang et al., 2021)[4] ■ Self-Supervised Policy Adaptation during Deployment (Hansen et al., 2021)[3]

List of selected paper

1. Unsupervised Reinforcement Learning

When solving sequential decision-making problems in real-world settings, it is often difficult to define rewards or they are too abstract even if they are defined. For example, to train robotic arms to pick up an object, the simplest way to define rewards is by assigning binary values for the outcomes: 1 if the robot successes 0 else. However, even though it looks like an easy task, it is difficult to learn for a robot that learns by such a simple reward. This is because the reward is only clear for the final result, and it is not informative feedback for the specific actions while the robot learns. If a robot has to perform a complex task, such as brewing hand drip coffee with Kenya AA beans, and if it still receives sparse rewards for pass or fail, it will be almost impossible for the robot to learn. One of the ways to mitigate this problem is to effectively design the feedback on the solving process as a reward function by humans (reward engineering). However, finding a good reward function that enhances learning becomes more difficult as the task becomes more complex, and even if we do find one, it becomes subordinated to the domain knowledge and has limitations of unable to extend to other similar problems. Therefore, it is not the direction we should be heading in, considering the recent trend in AI research that seeks to minimize human knowledge or labor.

In the field of reinforcement learning, an extensive amount of studies is underway to resolve cases where the information on rewards for learning is insufficient, and unsupervised reinforcement learning is out of them. As the name suggests, unsupervised reinforcement learning aims to enable an agent (e.g., robot) to learn meaningful behaviors without any task-specific reward. Although the description as to what is being learned is ambiguous, if an agent that has learned without any rewards performs a useful behavior that would have been done when performing a certain task, rather than repeating simple and random behavior, this alone carries an important meaning as the robot would be able to learn something on its own. In addition, these learned behaviors play a crucial role as they make it easier to learn complex tasks in environments given pass/fail - binary rewards, similar to unsupervised pre-training techniques before fine-tuning networks in early deep-learning studies.

Then, how can we make a robot learn useful behaviors without any rewards? Many studies are underway to answer this question, and the key idea underlying ??these studies is to motivate the robot to achieve more fundamental learning. For example, curiosity can be an important motivation for learning something, and encouraging behaviors to satisfy curiosity allows the robot to learn meaningful behaviors during this process. Certainly, there could be various motivations for learning other than curiosity and there could be a diversity of studies depending on how to motivate agents inspired by brain science, cognitive psychology, or allowing agents to effectively overcome the inherent characteristics of the given environment. I have selected two studies on unsupervised reinforcement learning introduced at ICLR 2021 and summarized which learning motive was suggested in each paper as below.


■ Mutual Information State Intrinsic Control (Zhao et al., 2021)^[5]

Motivation

“Mutual Information State Intrinsic Control (MUSIC)” proposed the unsupervised learning technique applicable to an environment where robots and objects interact, such as picking up or moving objects. The author presented an objective function that enables learning by dividing the robot (agent) state from its surrounding state to increase the correlation between the two, based on the fact that humans and animals can distinguish and recognize their own body and their surroundings. Therefore, this technique applies to an environment where the agent is observed being explicitly separated from the surrounding state as a separate vector.


Idea

The authors of the paper proposed a learning method that maximizes the mutual information (I) between the agent state (S^S) and the surrounding state (S^α) as shown below.

In this case, since I that measures the correlation between the two random variables is calculable only if the explicit probability distribution of each random variable is known, the paper used MINE (Belghazi et al., 2018)^[1] that estimates approximate values using a neural network. Therefore, as shown in the figure below, the agent learns a policy network that determines behavior and a neural network that produces mutual information, respectively. In particular, the policy network can learn simply by using the aforementioned I_Φ (S^S;S^α) instead of any rewards.

MUSIC learning algorithm

Experiment Results

It is confirmed how effective the proposed unsupervised training technique is when small rewards under the situation like pass/fail are given in a simulation environment for various interaction tasks between robot arms and objects. In this experiment, the state of the robot arm was defined as S^a and the coordinates of the target object was as S^s. The case in which fine-tuning is used as the rewards for a task after pre-training according to how MUSIC is utilized (-f), the case where prioritized experience replay (PER) based on I (-p) and the case where learning takes place using I and the sum of rewards were compared to the performance of the base model (DDPG and SAC). Among them, the case where I and the sum of rewards were used (-r) displayed the best performance, but it is worthy to note that PER, which repeatedly trains high I without using any rewards, displays significant performance. In the end, the paper confirmed that behaviors with a high correlation between the agent and the target object were useful for tasks performed for objects, and such learned behaviors are of great help when performing a certain task. 

Performance comparison based on how MUSIC is utilized in an environment with little rewards

Conclusions and Limitations

MUSIC showed that meaningful behaviors can be learned by maximizing the mutual information between the robot arm and the object that serve as the principal of behaviors in an environment where those two are separated. If the status is given as an image, it may be difficult to apply since the agent and the object are mixed together and one cannot be distinguished from the other. It would be interesting to consider the method of unraveling and learning the subject and the object mixed as a single vector through semantic segmentation or object detection on its own.

■ SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (Berseth et al., 2021)^[2]

Motivation

SMiRL assumes a specific problematic situation and offers an unsupervised reinforcement learning method. The authors suggested motivation to struggle for maintaining homeostasis against changes in an unstable environment, in which states vary continuously. It would be easy to understand by comparing the situation where failure occurs if the agent fails to cope with environmental changes to the game of Tetris. In Tetris, blocks continue to descend from the top of the screen, and if you fail to deal with them properly, you end up losing the game. Therefore, the authors believe that training an agent to maintain its status as static as possible can allow the agent to perform meaningful behaviors in an unstable environment.


Idea

To minimize the instability of the environment, the authors first used the entropy state as a measure of the degree of instability. From a probability distribution perspective, entropy is highest when all states have the same probability, and the greater the probability of a particular state, the lower the entropy. Therefore, minimizing entropy allows the agent to learn behaviors that allow it to remain in a constant state.

Where d^π_Φ(s_t) is the marginal distribution under its current policy π_Φ, because it is difficult to arrive at exact computation, the distribution  p_θt-1(s_t) is used instead, which is the distribution of collected states during a training phase. In SMiRL, the agent learns using  logp_θt-1(s_t) as the reward function, but if it is difficult to obtain the probability distribution, we can use VAE that approximates the probability density using the neural networks. In order to increase the likelihood of the observed state when the objective function is viewed from the VAE perspective, the agent must make sure the VAE does not experience unfamiliar states to the greatest extent possible. In other words, the learning should take place by maintaining homeostasis.  

SMiRL Learning Overview

Experiment Results

To verify the effectiveness of the proposed technique in an unstable environment, the authors chose Tetris and the shooting game, vizDoom. Like the nature of Tetris that continues to change its state, the authors also considered a scenario in which the opponent approaches and attacks first in vizDoom. The results of the experiment below show higher performance in Tetris and DefendTheLine scenarios when learning takes place using SMiRL without any rewards, compared to learning with rewards granted by the actual environment. This means reducing entropy is a more suitable feedback for solving the problem than any rewards defined by humans. In addition, even if the rewards provided by the environment are insufficient to train the agent, improving the performance using the objective function together is also found in other scenarios. (Note that whether the y-axis of the graph is the-smaller-the-better characteristics / the-larger-the-better characteristics)

Comparison of SMiRL’s learning performance in an unstable environment

The results of the second experiment demonstrate that SMiRL, which pursues homeostasis, and an objective function, which seeks novelty, perform more challenging tasks when used together. In the figure below, the red triangle must avoid the blue circle and move, and the environment ensures safety if it passes through the door in green.

In this situation, if learning takes place only with SMiRL, which only pursues homeostasis, the triangle is unable to pass through the green door and continues to be chased by the blue circle. However, the combined use of the objective function that pursues novelty, which contradicts the purpose of SMiRL, reinforces the exploratory behavior of the agent and derives better behavior. This is verifiable by comparing the behaviors of the agent according to the time below and the reward function logp_θt-1(s_t) at that time. In addition to the aforementioned results, how the agent trained under various environments behaves is shown via the link below: (SMiRL - Google Sites: https://sites.google.com/view/surpriseminimization/home)

Effectiveness of the combination of the preservation of homeostasis and exploratory tendency

Conclusions and Limitations

SMiRL suggests a technique to minimize the entropy state to train the behavior of preserving homeostasis in an unstable environment. In an unstable environment suggested by the authors, even though the agent behaved well as intended but if it still needed to learn the optimal behavior in a longer-term perspective, it was not sufficient to merely maintain homeostasis, and an additional objective function was needed to encourage the agent to explore. Although not covered in the paper, a balance between two objective functions that conflict with each other depending on the environment could be an issue. One of the interesting comments discussed in the Open Review pointed out that for a robot to learn to maintain homeostasis, a situation may unfold where it turns off the light in the room to shut out visual information by itself. Considering such issues, we need to give much thought into the objective function to determine what the agent should pursue.

 

2. Reinforcement Learning with Self-Supervision

In recent years regarding reinforcement learning, many studies have been conducted on the assumption that the agent observes pixel images under various situations, including the issue arising from robot control. If the early studies had mainly used vectors in a well-defined state, such as angles of robot arm joints, more recently, an extensive amount of studies are underway to learn the behavior with images composed of pixels using computer vision technology, similar to the way human beings behave after recognizing their surroundings with visual information. Pixel images make it easier to define various tasks in real-life settings. For example, as for a task of moving objects with a robot arm, information such as the location of a robot arm and the degree of fingers bent, and the location of objects is not defined one by one, but images alone enable learning the surroundings including the robot arm and objects filmed with cameras. However, the easier it has become to define a problem, the harder it has become to learn. First, since the image contains a mix of information useless for learning a task, including the surrounding background, it is difficult to identify what information is the key required for learning. In addition, since the policy network, which receives each state in real time to determine the behavior, is connected in the convolutional neural network and end-to-end methods to process images, and thus becomes massive in size, it is difficult to learn with a small amount of data or experience.

Learning a large-structured neural network properly is one of the main concerns in the field of machine learning that is subject to vibrant research to date. Several solutions have been suggested, but the most effective solution was the model that has been trained well in advance in many studies. As for computer vision, in particular, the convolutional neural network that has learned with ImageNet data composed of over one million images is most widely used. This approach would have been effective because the principle of extracting and processing information from pixel images with a two-dimensional lattice structure is similar in itself. However, there is no guarantee that it will still work well where a certain task or data differs significantly from the ImageNet data. To overcome these limitations, Self-Supervised Learning (SSL) has been suggested to ensure that the input data solves a new task using the data itself to resolve the task that we seek to address. For example, in principle, if the agent properly performs the task of distinguishing whether the two images that have been modified by various means are the same images or not, it will also be able to properly perform other tasks using the said images. SSL research mainly deals with the method of enlarging an image and how to define groups of the same images and groups of different images.

SSL is also effective for reinforcement learning issues combined with computer vision. The function of identifying the common features or differences found in two different images is the key to the tasks dealt with in computer vision such as image classification and object detection, and studies are widely conducted to determine which function helps reinforcement learning. I have selected two SSL research papers used in reinforcement learning introduced at ICLR 2021 and summarized how each paper defines SSL tasks.

■ Learning Invariant Representations for Reinforcement Learning without Reconstruction (Zhang et al., 2021)^[4]

Motivation

While driving a car, we accept many pieces of surrounding information with our eyes, but the essential information would be the car and road information. We do not need to focus on many sights around us, and paying too much attention would lead to an accident. With this in mind, the research to be discussed here focuses on how to learn to distinguish necessary and unnecessary parts on its own for the agent to perform a task in the state given in the form of an image. If the agent has learned well as intended, even if there is a difference in the surrounding background or small detail as shown in the figure below, the similar states, from the perspective of driving a car, are located closely in latent space. On the other hand, if two states are very similar in unnecessary parts such as backgrounds or minor details, but the core parts are different, they would be far from each other in the latent space.

Idea

This study proposed an encoder that preferentially selects information highly related to the task in a state composed of pixels and expresses it in the latent space. A well-trained encoder will be unable to tell us which part of the image constitutes a core area, but it will be able to distinguish between similar and non-similar states on its own based on the similarity in the core area. Therefore, the authors used SSL to train the ability to distinguish between similar and non-similar states. In particular, the authors used Bisimulation Metrics to measure similarity between two states in terms of reinforcement learning, not instead of training the similarity of pixels by using the image augmentation that is widely used for computer vision.

In the above definition, W₁ measures the distance between the two probability distributions with the Wasserstein Metric. This definition tells us that calculating the distance between any two states is not based on the similarity in pixels but how different the rewards were and how different the state of transitions was at the time. Based on this, the paper proposed a neural network to determine the distance of any two states as below. Although it may seem complex, its structure is similar to the Siamese neural network with Encoder - Dynamic model - Reward model that exist in pairs. One of the major differences is that the Reward model and Dynamic model are added respectively to obtain the rewards and the state transition probability distribution required for calculating the Bisimulation Metric. The policy that determines the behavior of the agent is used for input of the Dynamic model to predict the following states as the below architectures:

Encoder structure

Actual learning is carried out in the order of Policy, Encoder, and Dynamic model. Among them, Policy uses Soft Actor Critical (SAC), which learns using vectors, which have been transformed from the original image through the Encoder.

Experiment Results

If the limited technique had singled out the core area related to the task in the state as the form of an image, it would be able to properly learn in the background that continued to change, not in a static background. The figure below shows an example image of a 3x3 environment using DeepMind Control Suite and the results of nine experiments in response. The top row used the original environment with a static background, but the method suggested by the authors does not vary much from Reconstruction or other SSL-based methods. However, as the background continues to change and the change grows more unpredictable (the middle row is a predictable background transition and the bottom row is an actual video completed unrelated to the task), the technique suggested by the authors outperformed other methods. It is noteworthy that it achieved great performance without using autoencoders that require large-sized models for reconstruction.

Performance comparison of different techniques based on the degree of background change

The second result to be introduced is an experiment in which a comparison was made to determine whether the Encoder suggested by the authors considered only important information and carried out mapping properly in the latent space. Here, we can see that the behavior in the middle is the same, but the other two states are almost at the same location as the Encoder on the left suggested by the authors, but this is not the case for the VAE on the right. This is because VAE does not identify important information and only does mapping with the similarity in pixels.

Conclusions and Limitations

If a state is expressed using images, it is inevitable for the core part to be mixed with non-core backgrounds. The authors suggested a technique to select and learn important information necessary for performing tasks through the Encoder capable of distinguishing them. The technique suggested in a situation where key information may be diluted as the background information changes frequently may be well-learned but it demonstrates somewhat less performance compared to other techniques in a static situation. In other words, there is room for improvement for the suggested technique to also properly learn in a static environment. In this way, if the performance varies to a great degree by situation, such as like now, it can overcome the limitations in which a person must make decisions on the background change and determine the learning method.


■ Self-Supervised Policy Adaptation during Deployment (Hansen et al., 2021)^[3]

Motivation

When applying a model well-trained with reinforcement learning to a test environment, it often fails to perform adequately if there is any difference from the environment in which it was trained. In particular, in reinforcement learning using images, changes in areas with little importance, such as the background, serve as major obstacles to success in the test environment. This paper deals with a similar problem to the one previously discussed. However, this paper studied techniques for a well-trained agent to rapidly adapt to the test environment with changes in the areas unrelated to the task, such as the background. “Adapt” here refers to additional learning in a test environment.

Idea

In order to render the well-trained agent to rapidly adapt to the changed environment, an SSL module has been added in addition to the general reinforcement learning as shown in the figure below. The added SSL module helps the agent adapt by enabling it to continuously learn the state change that it has never experienced before in the test environment without any separate reward functions. (Additionally, it does not lead to any change to the task to be solved in learning and testing.) Even if there is a new change in the environment, such as the background, since there are parts that do not change, including the dynamics of the environment, it suggested the SSL technique to enhance learning. 

How to adapt to learning/test environment using SSL

In order to train the dynamics of the environment, the authors defined the following loss function to predict the sequence s_t created in the order of {s_t, a_t, s_t+1} and the behavior a_t that results in the change in the state that leads to s_t+1.

 π_e in the above formula shares the policy and SSL module that learns with reinforcement learning with the neural network that processes images with Feature extractor. An image that has been rotated by 90 degrees to train robust image processing was entered and an SSL task that can identify the degree of rotation was added to conduct the experiment.


Experiment Results

To verify how vulnerable well-trained reinforcement learning is to background changes or noise and how the SSL technique suggested by the authors can overcome the vulnerability, an experiment was conducted with a simulation that controls the robot arms and an actual robot. The figure below shows training the task of moving a yellow box to the red dot in the simulation environment to see whether the robot actually operates well. However, if a line is drawn on the table like a go board or a bright light is shone on the table, the robot does not operate well with the previous learning method. However, we can see that the suggested technique that allows continuous learning with SSL to adapt to the new environment enables the robot to operate well even in this situation. The experiment results are available in the video on Nicklas Hansen’s website  (https://nicklashansen.github.io/PAD/).

The results of the second experiment verified the suggested technique in the Nonstationary situation, where the background changed constantly, rather than in a static manner. The results are also shown in the same video on the website above. In this environment, the background continues to change, causing difficulty for the agent which had been well-trained in the original environment. Here, compared to other techniques that utilize SSL purely from the perspective of computer vision, like an image rotation, the suggested technique that also learns the dynamics in the environment, which serves an important role, was effective.

Conclusions and Limitations

This paper suggested an SSL technique to learn the dynamics of the environment to overcome the limitations that reinforcement learning using images fails to achieve its original performance in a test environment at variance with the learning environment. In fact, I read the paper as its ideas and experiment results were intriguing to me, but I could not come up with any limitations. However, to add my views from the perspective of further expanding this study, since it has dealt with the situation with the same task so far, applying the idea of Meta Learning to a different situation may produce further outcomes. 


One Step Closer to the Real Problem

As many problems found in the real world can be expressed as those arising from sequential decision-making, reinforcement learning applies to a wide variety of fields, but it is difficult to find any cases where it has been successfully applied to industries capable of creating much-added value. This is because various challenges lie everywhere for real-life problems, and they are tangled up in combination. Therefore, it is not easy to solve everything at once with a simple application. Therefore, it is meaningful for researchers to break down each problem into smaller units and derive limited but useful solutions, and this process will lead us to clues to solving real-life problems.

In this review, I reviewed papers on two research topics of unsupervised reinforcement learning and reinforcement learning with self-supervision under the common feature that the study on reinforcement learning is necessary to resolve real-life problems. I identified meaningful solutions derived from limited problematic situations that are covered in each paper, but at the same time, I was able to consider the foreseeable limitations. I hope that the limitations on the existing research will serve to be the starting line for additional research, and I am determined to remain further committed towards my research. At LG AI Research, researchers always encounter a host of real-life problems from our affiliates of various industries. Based on such environmental advantage and what I felt and learned at the ICLR 2021 conference, I will remain committed to contributing towards research on reinforcement learning with a step closer to real-life problems.