[NeurIPS 2023] Large Language Model-Based Autonomous Agents

1.  Summary

With the advent of OpenAI GPT-4 in 2023, large language models (LLMs) have become a huge success in many areas beyond AI. Reflecting this, NeurIPS 2023 saw a number of research presentations on LLM-based autonomous agents (LLM agents)^[1], which take advantage of LLMs’ vast prior knowledge and strong generalization capabilities.

An autonomous agent is a computing system that can interact with its environment and autonomously perform actions that have a predicted, intended outcome for some state or event. This post will introduce LLM-based autonomous agent research introduced at NeurIPS 2023, divided into (1) general-purpose LLM agent research, (2) domain-specific LLM agent research, and (3) ongoing AI agent research at AI Research.

2. General-Purpose LLM agents

Research on general-purpose LLM agents focuses on leveraging LLM to solve complex sequential decision-making problems that require reasoning and planning. At NeurIPS 2023, a number of interesting ways to address the limitations of traditional general-purpose LLM agents were presented.

2.1 Problem Solving

Figure 1. Tree of Thoughts^[2] conceptual diagram

Tree of Thoughts (ToT)^[2], selected as an oral paper at NeurIPS 2023, proposes a new prompting framework that extends the existing Chain of Thought (CoT) to enable LLMs to be used for general problem solving. Traditional methods using LLMs are primarily left-to-right, linear decision-making processes. Unlike these methods, the Tree of Thoughts proposes a way for LLMs to self-evaluate and search through multiple possible reasoning paths (see Figure 1). For example, for the computational reasoning problem “Game of 24”, the traditional CoT method has a success rate of only 4%, while the ToT method achieves a 74% success rate.

2.2 Linguistic Feedback

Figure 2. Reflexion^[3] conceptual diagram

Current LLM agents have the advantage of being able to achieve very high performance with very few samples, but there has been a relative lack of performance improvement through trial and error. Reflexion^[3] proposes a prompting framework that uses linguistic feedback to enhance the performance of LLM agents, rather than updating the weight of a neural network (see Figure 2). Reflexion refines the action trajectory by incorporating natural language feedback and reiterates multiple times to improve the performance of the action.

2.3 Adaptive Planning

Figure 3. AdaPlanner^[4] conceptual diagram

Currently, LLM agents are limited to performing actions without planning or relying on static plans to perform actions. AdaPlanner^[4] proposes a prompting framework for adaptive planning that allows LLM agents to self-improve the plans they generate in response to feedback from the environment (See Figure 3). To mitigate LLMs’ hallucinations, AdaPlanner uses code-style prompts. Then, through the skill discovery method, useful skills are used for the few-shot prompt.

2.4 Cooperation

Figure 4. SwiftSage^[5] conceptual diagram

SwiftSage^[5] chosen as a Spotlight paper, proposes an agent framework inspired by dual-process theory in cognitive science for action planning for complex reasoning tasks. SwiftSage consists of the Swift module, which is responsible for fast and intuitive thinking, and the Sage module, which is responsible for the deliberate thought process. The Swift module was implemented as a small Encoder-Decoder LM that learned the action trajectory of the oracle agent. The Sage module uses LLM like GPT-4 for subgoal planning.

3. Domain-Specific LLM Agents

In addition to general-purpose algorithms for LLM agents, many studies on domain-specific LLM agents were also presented at this conference. Notable domain-specific LLM agents include web agents that perform complex web browsing, coding agents that write executable code, embodied agents that perform tasks while interacting with virtual and real physical environments, and research agents that design, run, and analyze experiments for scientific discovery.

3.1 Web Agents

Figure 5. Mind2Web^[6] conceptual diagram

A web agent is an agent that performs complex tasks to manipulate a website by following commands given linguistically. Mind2Web^[6] was released as the spotlight in the Datasets and Benchmarks track. Mind2Web provides a dataset for developing and evaluating web agents (see Figure 5). Existing datasets are not suitable for developing real-world web agents as they are either simulation-based virtual websites or cover only a limited set of tasks. Mind2Web provides action sequences for over 2,000 tasks collected from 137 websites covering 31 fields. In addition, a general-purpose web agent called MindAct has been developed based on Mind2Web data to provide basic performance.

3.2 Coding Agents

Figure 6. InterCode^[7] conceptual diagram

A coding agent is an agent that writes code by following commands given linguistically. InterCode^[7] was presented in the Datasets and Benchmarks track in coding agent-related research. InterCode is a flexible and easy-to-use reinforcement learning (RL) environment for developing interactive coding agents (see Figure 6). InterCode is a programming language and is platform-independent. To emphasize the function and usability of InterCode, the authors of the paper set up an interactive coding environment for Bash, SQL, and Python. Different prompting methods are applied to the latest LLM, and the performance of each method is measured and reported on.

3.3 Embodied Agents

Figure 7. DEPS^[8] conceptual diagram

An embodied agent is an agent that interacts with a real or virtual physical environment to accomplish a goal. At NeurIPS 2023, a number of LLM-based embodied agents were introduced. One of them, DEPS^[8], proposes an interactive planning method called DEPS (Describe, Explain, Plan, and Select) to effectively achieve goals in a cyber-physical environment such as Minecraft. This method allows the initial plan generated by the LLM to be improved with additional explanations of the execution process, feedback on execution errors, and the selection of degree goals based on predictions. With this method, the paper shows that more than 70 tasks in Minecraft can be achieved with very high performance.

3.4 Research Agents

Figure 8. MLAgentBench^[9] conceptual diagram

A research agent is an agent that performs experimental design and analysis for scientific discovery. Regarding research agents, MLAgentBench^[9] was announced at the Foundation Models for Decision Making Workshop. MLAgentBench is a type of reinforcement learning (RL) environment for evaluating the performance of research agents performing linguistically given machine learning (ML) tasks (e.g., Improve the performance of the image classification model that “train.py” builds) (see Figure 8). Agents can perform actions such as reading/writing files, executing code, and analyzing results. Also, the performance of the agents is evaluated through execution results (Submission.csv file) and interaction trajectories.

4. LLM Agent Research by LG AI Research

LG AI Research is also conducting research on LLM agents and presented several research papers on the subject at NeurIPS 2023. Here are a few papers on LLM agents published by LG AI Research.

4.1 MultiPrompter

Figure 9. MultiPrompter^[10] conceptual diagram

In-Context Learning (ICL)-based LLM agents are highly dependent on the prompts used for performance. MultiPrompter^[10] proposes a method to find the best prompts automatically, rather than requiring a human to do prompt optimization every time the foundation model is used. In particular, a new perspective on the traditional prompt optimization problem was proposed: multiple agents collaborating to optimize together. With this collaboration, the prompt search space was effectively partitioned, and confirmed that significantly improved prompts were obtained compared to existing methods. This paper was selected as a Spotlight at the Efficient Natural Language and Speech Processing (ENLSP) workshop.

▶MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning (Link)

4.2 Prospector

Figure 10. Prospector^[11] conceptual diagram

The problem with current LLM agents is that they are primarily based on simple, few-shot demonstrations, which limits their ability to select and evaluate the optimal action to complete a given task. To address these issues, Prospector^[11] proposes an agent consisting of two complementary LLMs: an LLM Actor and an LLM Critic.

An LLM Actor can create better actions through a prompting method called self-asking. The LLM Critic is responsible for predicting the rewards of different trajectories generated by the LLM Actor. Through trajectory ranking, which selects trajectories that are predicted to have higher rewards, Prospectors increase the success rate of their assignments. Through a household activity simulation environment called ALFWorld, and an online shopping simulation environment called WebShop, it is experimentally shown that the proposed LLM agent can achieve a higher success rate than existing LLM agents such as ReAct and Reflexion.

▶Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking (Link)

5. Conclusion

At NeurIPS 2023, various workshops related to LLM agents were held with great success. Among these, prominent workshops include Foundation Models for Decision Making (FMDM)^[12], Agent Learning in Open-Endedness (ALOE)^[13], Instruction Tuning and Instruction Following (Instruction)^[14]. Each workshop presented interesting research and methodologies related to LLM agents, and researchers working on them were able to share their latest trends and insights.

So far, we divided the latest research on LLM agents introduced at NeurIPS 2023 into (1) general-purpose LLM agent research, (2) domain-specific agent research, and (3) LG AI Research’s agent research. LG AI Research is actively researching various LLM agents based on EXAONE, a large-scale AI. For example, agents capable of complex reasoning, such as solving math problems using external tools, and agents capable of adaptive planning based on feedback from the environment or the user are currently being worked on. We expect that this research will enable us to effectively develop more advanced domain-specific LLM agents.

▶ Want to learn more about the NeurIPS 2023 Research Blog series? (Link)