Geonhyeong Kim 2023.01.06

[NeurIPS 2022] LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation

This post will introduce a paper accepted at NeurIPS 2022 by Geonhyeong Kim, who joined LG AI Research in July. Kim conducted this research in the lab of Professor Keeeung Kim at KAIST. The lab of Professor Kim conducts numerous studies on reinforcement learning (RL), including fundamental theories of RL and imitation learning (IL), as well as RL combined with natural language. This paper focuses on the fact that humans learn by observing state-only expert demonstrations and proposes a learning from observation (LfO) method in an offline setting.

Introduction

The ability to learn from experience is one of the core aspects of an intelligent agent. RL is a framework in which an intelligent agent is trained through interactions with the environment while receiving reward feedback. However, since designing an appropriate reward function for complex tasks can be extremely difficult, IL utilizes expert demonstration to teach without a reward function on how to imitate the expert's behavior. Whereas this general IL requires behavioral information in addition to expert state information, humans learn without such behavioral information when they are learning to drive or learning by watching videos.

Figure 1. State-only expert demonstration with no action labels when learning to drive

As such, LfO is a problem in which the aim is to mimic the expert policy from the state-only expert demonstrations. However, existing LfO problems necessitate interaction with the environment, which is often risky or costly.

Figure 2. A conceptual diagram of a problem setting involving offline LfO

Instead of interacting with the environment, an offline LfO that uses imperfect demonstrations safely collected in advance was proposed. Existing methods to this problem include the off-policy LfO algorithm^[1] and a method for extending a general offline IL algorithm using an inverse dynamics model^[2]. In the case of an off-policy LfO algorithm, however, offline learning is unstable due to its use of an out-of-distribution (OOD) action value and a max-min optimization. The inverse dynamics model is expressed by the following formula; when it has a stochastic transition probability, it can vary for each policy even in the same environment.

d_{I}^{I} (a s, s') = \frac{d^{I} (s, a, s')}{d^{I} (s, s')}

Predictions of expert behavior in a stochastic environment become inaccurate in the case of an inverse dynamics model learned from imperfect demonstrations. Therefore, there is a performance limit, even if the existing offline IL algorithm is extended by using the inverse dynamics model. In this paper, we propose a new formula-based algorithm for achieving optimal performance while resolving offline instability.

Method

In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from the data support) is one of the common approaches to avoid severe performance degradation. For the same reason, we use KL divergence minimization between ${\bar{d}}^{π} (s, s')$ and ${\bar{d}}^{Ε} (s, s')$ with additional KL regularization:

\underset{π}{m i n} D_{K L} ({\bar{d}}^{π} (s, s') ∥ {\bar{d}}^{E} (s, s')) + α D_{K L} (d^{π} (s, a) ∥ d^{I} (s, a)) .

Here, α>0 is a hyperparameter that balances the state-transition matching and preventing distribution shift from the distribution of imperfect demonstration. Rewriting the distribution matching problem in terms of optimization of stationary distributions yields the following:

\max_{d, \bar{d} \geq 0} - D_{K L} (\bar{d} (s, s') ∥ {\bar{d}}^{E} (s, s')) - α D_{K L} (d (s, a) ∥ d^{I} (s, a)) s . t . \sum_{a'} d (s', a') = (1 - γ) p_{0} (s') + γ \sum_{s, a} d (s, a) T (s' s, a) \forall s', \sum_{a} d (s, a) T (s' s, a) = d (s, s') \forall s, s' .

$d (s, a)$ is a valid state-action stationary distribution for some policy π due to the first constraint, commonly referred to as the Bellman flow constraint. The second constraint, the marginalization constraint, enforces $\bar{d} (s, s')$ to be the state-transition stationary distribution that is induced by $d (s, a)$ . We then consider the Lagrangian for this constraint optimization:

\min_{μ, v} \max_{d, \bar{d} \geq 0} - E_{\bar{d}} [l o g \frac{\bar{d} (s, s')}{d^{E} (s, s')}] - α E_{d} [l o g \frac{d (s, a)}{d^{I} (s, a)}] + \sum_{s, s'} μ (s, s') (\bar{d} (s, s') - \sum_{α} d (s, a) T (s' s, a)) + \sum_{s'} ν (s') ((1 - γ) p_{0} (s') + γ \sum_{s, a} d (s, a) T (s' s, α) - \sum_{a'} d (s', a')) .

Here, $v (s) \in ℝ$ and $μ (s, s') \in ℝ$ are Lagrangian multipliers for the Bellman flow constraint and the marginalization constraint, respectively. We rearrange the above optimization problem as follows:

As $x \log x$ is strict convex, we can obtain a closed-form solution of the inner maximization. Substituting the closed-form solution $(w_{μ, v}, {\bar{w}}_{μ})$ into $ℒ (w, \bar{w}, μ, v)$ , the following is obtained:

\underset{μ, v}{\min L} (w_{μ, v}, {\bar{w}}_{μ}, μ, v) = (1 - γ) E_{s ~ p_{0}} [v (s)] + E_{(s, s') ~ {\bar{d}}^{1}} [\exp (r (s, s') + μ (s, s') - 1)] + α E_{(s, a) ~ d^{1}} [\exp (\frac{1}{α} e_{μ, v} (s, a) - 1)]

This method yields a convex minimization without OOD action value. Lastly, the problem of extracting the optimal policy from $(μ^{*}, v^{*})$ obtained by solving the preceding optimization problem remains. For this, we adopt weighted behavior cloning (WBC) as follows:

\underset{π}{\max E} {}_{(s, a) ~ d^{π *}}[l o g π (a s)] = E_{(s, a) ~ d^{I}} [w_{μ *, v *} (s, a) l o g π (a s)]

Experiments

Experiments were conducted on tabular MDP and continuous MDP to evaluate LobsDICE. In Tablular MDP, the following four baseline algorithms were used: BC, BCO,^[2] and OPOLO^[1] in imperfect demonstration, as well as an algorithm dubbed DemoDICEfO designed by us. Here, DemoDICEfO is an extension of the state-of-the-art offline IL algorithm DemoDICE^[3]. DemoDICEfO trains an inverse dynamics model, uses it to fill the missing actions in the expert demonstrations, and then runs DemoDICE using the approximate expert demonstrations and the imperfect demonstrations. For continuous MDP tasks, we use two additional baselines: IQ-Learn^[4] and RCE^[5].

We first experimented with LobsDICE and baseline algorithms in randomly generated MDP, varying the number of expert/imperfect demonstration trajectories and the degree of stochasticity of transition probabilities. In the highly stochastic MDP (β=1 in Figure 3), LobsDICE asymptotically recovers the expert’s demonstrations while baseline algorithms are suboptimal.

Figure 3. Performance of LobsDICE and baseline algorithms on randomly generated MDPs. The total variation between the learned policy and the expert's stationary state-transition distribution was measured; greater performance is indicated by smaller values^[6]

LobsDICE and baseline algorithms were then evaluated in the MuJoCo continuous control task. Using the D4RL dataset, we constructed several tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant. LobsDICE was the only algorithm that consistently recovered the expert’s performance across a variety of tasks, thereby outperforming baseline algorithms.

Figure 4. The higher the performance in MuJoCo control tasks of LobsDICE and baseline algorithms, the greater the performance ^[6]

Conclusion

We have presented LobsDICE, a new algorithm suitable for offline LfO. LobsDICE shows stable and promising performance by learning through convex optimization without OOD action values. LobsDICE has experimentally achieved state-of-the-art performance on a variety of tabular and continuous tasks. By extending this work, we intend to develop an algorithm for an intelligent agent that can learn from a wider variety of data types.

참고

[1] Zhu, Zhuangdi, et al. "Off-policy imitation learning from observations." NeurIPS, 2020.

[2] Torabi, Faraz, Garrett Warnell, and Peter Stone. "Behavioral cloning from observation." IJCAI, 2018.

[3] Kim, Geon-Hyeong, et al. "DemoDICE: Offline imitation learning with supplementary imperfect demonstrations." ICLR, 2022.

[4] Garg, Divyansh, et al. "IQ-Learn: Inverse soft-Q Learning for Imitation." NeurIPS, 2021.

[5] Eysenbach, Ben, Sergey Levine, and Russ R. Salakhutdinov. "Replacing rewards with examples: Example-based policy search via recursive classification." NeurIPS, 2021.

[6] Kim, Geon-Hyeong, et al. "LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation." NeurIPS, 2022

목록보기