[NAACL 2022] Language Models for Planning in Real-world Tasks

We show that language models encode strong common-sense priors useful for planning a sequence of intermediate steps from text instructions. Using the in-context learning ability of language models, we predict plans with high recall (70%) given as few as 20 training examples. In addition, we show that predictions can be further improved using Mutual Information inspired ranking criteria and by incorporating interaction and feedback from the environment.

Introduction and Motivation

Consider instructing a household robot to perform a basic task like making a cup of coffee. The robot has to navigate around the house, interact with various objects and try to accomplish the task. This is a challenging task for the robot that involves multiple reasoning capabilities including visual understanding, text understanding, grounding, remembering information over many time steps, and so on. As humans, we approach these tasks by first planning at an abstract level and using this plan to guide our actions. If we can endow such planning capabilities in artificial agents, they would be more successful at performing real-world tasks.

Our goal is to use the strong common-sense priors learned by language models to reason about subgoals. We show that language models can

• reason about subtasks from task instructions with very little supervision
• reason about object affordances
• enable systematic generalization to unseen instructions involving unseen subtasks

Specifically, we focus on the subgoal planning problem. Given a high-level task directive, our goal is to break down the task into a sequence of subgoals or object interactions and build a modular agent policy that can perform the task in an embodied environment.

Subgoal Planning with In-context Demonstrations

We predict subgoals using an emergent property of language models called in-context learning (Figure 1). We provide a few demonstrations of the language model in the form of instructions paired with subgoal sequences. We then give it a query instruction and predict subgoal sequence hypotheses using a constrained beam search. The hypotheses are ranked based on the language model probabilities.

Figure 1. Subgoal prediction with in-context learning. Given a few demonstrations of instructions paired with subgoal sequences, the model predicts candidate subgoal sequences for a given query instruction.

Mutual Information based Ranking

Although language models can generate compelling hypotheses, we find that the ranking of these hypotheses is plagued by calibration issues^[1]. Consider the query instruction ‘Place a martini glass with a fork on it on the table’. In the top-1 model prediction, the model incorrectly predicts the target receptacle as ‘sink’ (instead of ‘table’). However, when we remove all training demonstrations that involve a ‘sink’ and have the model predict again, it now predicts the correct target receptacle as ‘dining table’. What we find here is that even though the language model can identify strong candidates, the ranking of these candidates is plagued by the so-called popularity bias^[2].

To address calibration issues, we consider ranking criteria inspired by mutual information. We consider a weighted version of the pointwise mutual information (eq 1) where q and h are the query and a hypothesis, and λ is a hyperparameter.

Equation 1

Maximizing the weighted PMI yields the following ranking criterion.

Equation 2

This is a linear weighting of two quantities.

• For λ = 0, we recover our previous ranking metric, which is the probability of hypothesis given query.
• For λ = 1, we obtain the probability of query given hypothesis as the metric. This is also called noisy channel prompting in the literature [3]. We can interpret this probability as how well the subgoal sequence explains the text instruction.

For intermediate values of lambda, we linearly combine these two metrics.

We evaluate this approach on instructions from the ALFRED^[4] benchmark. Tasks from this benchmark involve a sequence of object interactions such as pick and place, slicing, heating, and so on. Each instruction is annotated with subgoals computed by a planner, which we use for training and evaluation. There are 7 task types and 11 fine-grained types. We ignore the navigation subgoals and only consider the 7 object interaction subgoals such as pick up X, place X, etc. where X is an object argument that comes from a vocabulary of 80 object types.

Figure 2 shows a comparison of results for different ranking criteria. Ranking predictions based on p(q|h) leads to improvement over p(h|q) when using 1 demonstration per task type. But we don’t see the same benefits when more supervision is available. On the other hand, combining these two metrics with λ = 0.5 yields consistent improvements over both of these metrics and brings us closer to the top-10 recall ceiling.

Figure 2. Subgoal prediction accuracy of the GPT-J language model for varying amounts of supervision where we compare the performance for different ranking metrics.

Subgoal Planning with Environment Feedback

So far, we have considered planning from text instructions alone. However, we need to incorporate information from the agent’s environment to predict subgoals accurately. For instance, examining the model errors we find that the model is confused between object categories such as floor lamp and desk lamp. Unless the type of lamp is precisely specified in the text instruction, it is unlikely that the model is able to identify the correct category. Information acquired from the environment can inform the model about the type of tasks that can be performed (e.g., if you are in a kitchen you can perform tasks such as slicing, heating, and cooling) and what objects are available for interaction (e.g., a knife is likely to be found in a kitchen). Such information is useful for more accurate subgoal predictions.

Given a subgoal sequence, we first construct an agent policy using a pre-trained subgoal policy which executes the subgoals against the environment in sequence. The language model can be thought of as a high-level controller that provides goals for the low-level subgoal policy to execute. Given this agent policy, we can execute language model predictions against the environment and collect success/failure feedback. We then use this feedback to train a ranking model to re-rank the language model predictions based on the agent’s state representation, which is a vector representation that captures information about the environment.

During inference, we use this trained re-ranker to the re-rank model generated hypothesis at every step (Figure 3). In this example, the query instruction ambiguously specifies the target receptacle as ‘white table with shelving’. Even though the language model originally thinks that this corresponds to a ‘shelf’, the agent state representation suggests that a ‘dining table’ is present in the environment, based on which the correct hypothesis is chosen.

Figure 3. Language model hypotheses are re-ranked at every step of execution based on the agent’s state representation. Even though the original top-1 prediction is incorrect, the re-ranker identifies the correct subgoal sequence by using information from the agent’s state representation.

We evaluate this agent policy on the ALFRED benchmark. Unlike the previous evaluation where we looked at plan prediction accuracy, the metric here is the success rate of performing tasks in the embodied environment. MOCA^[5], FILM^[6] and HLSM^[7] are some recent approaches designed for this benchmark. We make use of the low-level subgoal policy from HLSM in our experiments.

We compare the different models in Figure 4. First, we look at the in-context learning result with PMI based ranking which uses only 20 demonstrations (No env feedback). With a re-ranker trained based on 1000 instances of feedback (With env feedback), we get an improvement, but this is still an order of magnitude less subgoal supervision compared to the HLSM model. Finally, we can also look at the performance with oracle subgoals, which represents the performance upper bound with the HLSM low-level policy. This shows that there is room for improvement, and improving subgoal predictions can lead to a better overall performance of the system.

Figure 4. We compare the task completion success rates of different approaches on the ALFRED benchmark. Our models (shown in green) make use of the low-level pre-trained policy from HLSM and we show the performance with/without environment feedback. With 1000 instances of success/failure feedback, our approach is competitive with the baseline policies, while using an order of magnitude less supervision.

Conclusion and Future Work

Our work shows that language models can help reason plan intermediate goals for performing real-world tasks. In particular, we show that in-context learning combined with large language models helps us generate compelling subgoal sequences with just a few demonstrations. We show that mutual information inspired ranking metrics can help better rank predictions from the language model. Finally, we show how predictions can be further improved by collecting feedback from the agent’s environment and training a ranking model.

Although we show that information about the environment can be incorporated into model predictions through the use of a ranking model, we believe that the language model can benefit from tighter integration with the agent policy and environment which can be explored in future work. It would be further beneficial to study ways in which the robustness of the agent policy can be improved such as recovering from planning errors.

▶Few-shot Subgoal Planning with Language Models (Link)