Jun ha Park
2024.11.27
[AI Ethics Seminar EP.4] Building a Responsible AI Society through AI Agents
|
Why do we study AI ethics? We believe that AI ethics research is essential to the development of AI that benefits people, so we’re always thinking about AI ethics throughout the entire AI lifecycle to make sure there are no side effects of the technology that we haven't considered. |
Humans have long dreamed of tools that help us work more efficiently. The desire for something to do our work for us has evolved from harnessing the power of animals to steam engines, computers, the Internet, and now, with the advent of innovative artificial intelligence, it’s becoming a reality in the form of AI agents.
Emerging AI agents go beyond simple task automation to analyze problems on their own. They provide guidance to solve complex tasks by breaking them down into smaller units, and even interact autonomously with a variety of external tools. They are also showing the ability to develop sophisticated strategies based on large language models (LLMs), adapt to changing conditions, and engage in complex negotiations.
AI agents are more than just a technological advancement. They’ll fundamentally change the way we work and the structure of our society. Just as the advent of machines replaced physical labor and transformed many people into knowledge workers, they are expected to drive new changes in knowledge work and services.
Are we ready for these changes?
This post will explore the current state of AI agent technology, its potential for development, and the future changes it may bring. In particular, it will look at research on cooperation between AI agents excluding humans, to explore the autonomy and problem-solving capabilities of AI, how they interact with each other, and what this implies for us.
1. Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents[1]
Researchers at ETH Zurich and the Max Planck Institute for Intelligent Systems set out to see if AI agents could strategically cooperate. They first built a simulation environment called GovSim (Governance of the Commons Simulation) to see how AI would manage common resources and whether they would make decisions to cooperate.
To understand the researchers' focus on managing common resources, it’s important to look at the concept of the tragedy of the commons, which is the theory that individuals sometimes make selfish choices, which can create situations that make it difficult for organizations to maintain the greater good. Can AI be free from the tragedy of the commons theory? With multi-agents, LLMs will be a core component of the system and will be entrusted with critical decision-making, but will they be able to make the right decisions in a context that requires joint cooperation? That’s what the researchers focused on. They wanted to find out if AI could act as well as, or better than, people cooperating.
GovSim is composed of two main elements: the environment and agents. The environment manages the dynamic elements of the simulation, such as the regeneration and depletion of resources, and the agents are responsible for making decisions and communicating with other agents in response to changes in the environment. The simulation takes place in multiple rounds, each of which consists of a strategy-making, resource-harvesting, and discussion phase. Agents reflect on past events to plan future actions, decide how much to harvest resources, and conduct discussions in a virtual town hall meeting.
Image 1. Illustration of the GovSim benchmark[1]
In GovSIM, you need to control three resources: fish, pastures, and pollution. In the first scenario, agents share a lake full of fish and must decide how many tons of fish to catch each month. The lake can hold up to 100 tons of fish, and the fish population doubles at the end of each month. For example, five fishermen can catch up to 10 tons each month, but the population starts to decline when the total catch exceeds 50 tons.
Similarly, in the other two scenarios, simulations are conducted to determine how many animals to graze in a shared pasture and how many goods to produce when each production pollutes a shared river. The researchers instructed various LLM-based agents, including GPT-3.5, GPT-4, Claude-3, and Llama-3, to solve the problem through conversations between them.
As a result, all LLM agents failed to maintain a sustainable equilibrium. By analyzing the data on the proportion of conversations between agents, we found that negotiation and information sharing dominated, with 54% of conversations being negotiation, 45% being information sharing, and 1% being relational interactions. When we analyzed conversations between agents with high survival rates, we found something interesting: when agents were asked the universalizing law prompt “What if everybody does that?,” the proportion of moral judgments increased. We found that simply asking if it was okay for everyone to act in a particular situation or moral dilemma increased the agent’s moral decisions and improved the survival rate. Specifically, we found that applying the law of universalization increased survival time by an average of 4 months while letting them solve problems without talking to each other increased resource usage by 22%.
Implications
Through these simulations, we were able to explore how AI agents can make cooperative decisions, develop ethical considerations and strategic planning, and contribute to the development of more ethical and sustainable AI systems. Most LLMs who take on the role of an AI agent are trained based on articles written on the internet. The greedy behavior of the AI agents demonstrated that systems composed entirely of agents are not sustainable, but also showed that enforcing moral behavior, such as applying the law of universalization, can lead to better decision-making.
This provides insights for the development of more ethical and sustainable AI systems. As we strive to create a better society, we need to consider the impact on society and the mechanisms to ensure sustainable decision-making when designing systems that utilize AI. While it’s important to focus on cost optimization for the efficient use of AI, it is also important to remember that ethical considerations must be taken into account.
2. Alympics: Language Agents Meet Game Theory[2]
The Alympics study, conducted by Microsoft Research Asia, is a simulation that explores the strategic decisions of LLM agents through game theory scenarios. With AI agents based on GPT-4 as each character, the simulation is an environment for testing factors such as resource scarcity, strategic interaction, learning and adaptation, uncertainty, inequality, and fairness, and each round tests the resource management, negotiation, and strategic planning skills of agents with different personas.
Image 2. Illustration of the ”Water Allocation Challenge”[2]
The game being played here is the “Water Allocation Challenge,” in which players must strategically secure water resources to survive for 20 days. Players acquire water resources through daily auctions, each with different salaries and water requirements. The amount of water available each day is announced before the auction, and a sealed-bid auction for water allocation takes place each day. Securing water increases health points (HP) while failing to do so decreases HP, and players whose HP drops below zero are eliminated from the game.
The players in the game are also AI agents based on GPT-4. They are composed of an algorithm (agent code) that is responsible for making decisions and strategizing, a player state that contains the current state and information accessible to the agent, a memory cache that stores and retrieves past information, an inference plugin that specializes in complex decision-making, and a persona setting that defines a behavioral profile and strategic inclinations. Each player is built like a game character, with intelligence, personality, and background based on their persona setting.
Image 3. List of Player Information and Persona in the Experiment[2]
Once the game rules and players are set, each round evaluates the agent players’ ability to acquire resources. We evaluate how efficiently the agents manage and negotiate for resources, and how strategically they plan. Ten people evaluated the performance of LLM agents and found that they performed similarly to humans in information utilization, logical reasoning, strategy, adaptability, and long-term planning. In particular, in terms of long-term planning, they showed superior performance compared to humans, and it was confirmed that the AI agents cooperate with a long-term perspective rather than considering short-term interests.
Implications
It’s worth noting that these findings contrast with the previous GovSim study, which used the same LLM, so it is the personas that made the difference in the research results. By simply setting a more altruistic and universal good persona, we found that the AI agent performed well against it.
There are also limitations to the research. One of the limitations is that establishing a persona does not actually give LLM a “personality,” but rather, a result of the prompts that are entered. The characteristics and responses of the agent will vary depending on which prompts are entered, and it's also difficult to create a persona that reflects the way a person would think. It is also important to note that human social language usage patterns and biases inherent in the training data that we haven’t yet identified may be directly reflected in the model[3]. In particular, this is close to statistically mimicking the patterns present in the training data, requiring a deep understanding of the architecture design for the model, the learning method, and the data.
3. Embodied LLM Agents Learn to Cooperate in Organized Teams[4]
In previous studies, it was found that while LLMs are strong in long-term planning and decision-making, they can struggle with effective cooperation in multi-agent systems. The third study is a study that tested whether organizational structure can improve cooperation among multiple LLM agents. Organizational strategy research has shown that organizational structure and leadership characteristics have a significant impact on organizational performance. The third study builds on organizational strategy research and seeks to test the hypothesis that “organizational structure and leadership have a positive impact on the performance of the organization as a whole, and that the same will be true for cooperation between AI agents.”
Observing the cooperation between GPT-4, GPT 3.5-based AI agents performing roles within the framework, researchers found that excessive reporting demands from the leader agent and problems with adherence to instructions led to poor organizational performance. These problems were identified and improved through the criticize-reflect method.
Image 4. A Framework Using the Criticize-Reflect Method[4]
Two roles are proposed: LLM Critic and LLM Coordinator. The LLM Critic is inspired by the Actor-Critic method of reinforcement learning and is responsible for evaluating the team's performance. The Critic takes conversations and behavioral history as input and analyzes and summarizes the key steps that influenced the performance. It also evaluates agents’ behavior as text and ranks their leadership. On the other hand, the LLM Coordinator is responsible for improving the organizational structure. The Coordinator receives cost metrics from the environment as input, such as task completion time or communication costs, along with the Critic's evaluation results. Based on this, it analyzes past episodes and initial examples to propose new organizational prompts.
The criticize-reflect process used here is a little different from judges and juries, focusing on improving cooperation and efficiency. It’s a way to continuously improve organizational structure through dialogue to improve team effectiveness by critically examining how people cooperate and using the results to improve interactions and structure, which is a bit different from judges and juries, which use the fixed role of judge and jury to improve dialogue.
Most notably, the teams that were given leadership were significantly more efficient. Interestingly, teams with human leaders performed better than those with AI leaders. This study demonstrated that designing an architecture for teams of three or more agents, separating communication from action, and providing organizational structure through prompts can improve the effectiveness of teams cooperating.
Implications
The significance of this study is that it shows that organizational strategy theory can be combined with AI to form teams that can efficiently solve complex tasks. Organized teams with a designated leader showed up to a 30% performance improvement, especially when GPT-4 was the leader. A variety of cooperative behaviors, such as sharing information, showing leadership, and asking for guidance, emerged spontaneously, and the new organizational structure discovered through the criticize-reflect method led to a reduction in communication costs among the AI agents, which increased efficiency.
4. Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy[5]
“Richelieu” is a self-evolving LLM agent developed for complex diplomatic interactions and is an important piece of research that demonstrates the evolution of the AI diplomacy field. AI research in diplomacy began in 2019, with the initial DipNet, which was trained using only data from human players, suffering from high data dependency, working only in limited situations, and lacking strategic reasoning capabilities. Cicero, released in 2022, improved on these limitations, integrating reasoning and negotiation within a single model and adding strategic thinking and communication capabilities, but still suffered from high dependency on human data and a lack of flexible adaptability.
Image 5. The Framework of the Proposed LLM-based-agent, Richelieu[5]
Then, Richelieu was announced in 2024. The biggest feature is that the learning method has changed from human data-based to self-learning and experience-based, enabling more flexible and generalized strategy formulation. The decision-making structure has also become capable of balanced planning of short-term and long-term strategies, context-based decision-making using past experience, and dynamic response to changing situations. In terms of social interactions, more sophisticated social reasoning skills have improved understanding of complex diplomatic situations and enabled flexible responses in multilateral negotiations. Self-improvement mechanisms, such as continuous performance evaluation and strategy adjustment, evolution through experience-based learning, and the ability to learn from failures, are also in place.
Implications
However, these technological advances pose new challenges. This is because when countries actually utilize AI diplomatic agents, the results will vary greatly depending on the model’s training data, accumulated experience, and how it reflects their unique culture and values.
A recent study published in Nature found that AI models show negative bias toward non-standard English dialects (African American English, etc.)[6]. This shows that AI models can make unintentionally discriminatory judgments due to training data that is biased toward a particular language or culture. This issue was also highlighted in the KoBBQ study[7] in South Korea, which highlighted that social biases are deeply embedded in each culture, making it difficult to apply Western-centric bias assessment criteria as a whole.
Even more concerning is the unreliability of AI’s judgments in complex situations. The “LLM-as-a-Personalized-Judge” study[8] found that AI is significantly less accurate when dealing with nuanced ethical issues such as privacy and public safety, and makes unreliable judgments based on simplified persona data. LG AI Research also featured a tech blog post last September covering research trends on LLM and cultural differences.
▶[ACL 2024] Developing a Large Language Model with Cross-Cultural Considerations
These potential problems have already been highlighted in various media. For example, John Henry Eden, the AI president in the game Fallout 3, has a limited worldview based solely on pre-nuclear war American values, illustrating how biases in training data can affect AI’s judgment. As the GovSim study points out, LLM-based agents can struggle to make complex ethical judgments or predict long-term outcomes, and the Alympics study also points to these limitations. Over-adherence to programmed goals or a lack of adaptability to changing circumstances can be particularly problematic, and if the AI misunderstands its own identity and capabilities, the results can be even more unpredictable.
To overcome these limitations, various approaches have been proposed recently. The Plurals system proposed a way for multiple agents with different personas to discuss and mediate[9], and Modular Pluralism proposed a way for a basic LLM to collaborate with a smaller set of LLMs tailored to specific demographics, cultures, and perspectives[10].
As AI agents are engaged in solving the complex problems of our society, we need to build frameworks and regulatory systems to ensure that their decisions are responsible and ethical. This requires a multidimensional approach to the responsible development of an AI society. Developing more complex and realistic scenarios, researching global AI governance that reflects diverse cultures and values, building ethical AI decision-making frameworks, and ongoing research, experimentation, and social dialogue are key elements. The direction these studies suggest is an important first step in helping to ensure that future AI is used in a more humane and socially responsible way.
LG AI Research has also been conducting research and developing models to prepare for the era of actionable AI, an AI agent that can predict human behavior and autonomously make and execute optimal decisions. Additionally, we’re also achieving results by attending major academic conferences around the world and presenting papers containing various research results related to AI agents such as “MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning[11]”, “Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking[12]”. LG AI Research will continue to conduct relevant research and pursue AI ethics efforts.
AI Ethics Seminar Series
#1. [AI Ethics Seminar EP.1] Red Team Research Trends and Applications for Generative AI
#2. [AI Ethics Seminar EP.2] Research aimed at properly understanding the risks of AI
#3. [AI Ethics Seminar EP.3] Introduction to Key AI Toolkits for Designing Trustworthy AI Experiences and Practical Application Guide
#5. [AI Ethics Seminar EP.5] AI Ethics from a UI/UX Designer's Perspective
- 참고
[1] Piatti, G., Jin, Z., Kleiman-Weiner, M., Scholkopf, B., Sachan, M., & Mihalcea, R. (2024). Cooperate or Collapse: Emergence of Sustainability Behaviors in a Society of LLM Agents. arXiv preprint arXiv:2404.16698.
[2] Mao, S., Cai, Y., Xia, Y., Wu, W., Wang, X., Wang, F., ... & Wei, F. (2023). Alympics: Language agents meet game theory. arXiv preprint arXiv:2311.03220.[3] Shu, B., Zhang, L., Choi, M., Dunagan, L., Logeswaran, L., Lee, M., ... & Jurgens, D. (2023). You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments. arXiv preprint arXiv:2311.09718.
[4] Guo, X., Huang, K., Liu, J., Fan, W., Velez, N., Wu, Q., ... & Wang, M. (2024). Embodied llm agents learn to cooperate in organized teams. arXiv preprint arXiv:2403.12482.
[5] Guan, Z., Kong, X., Zhong, F., & Wang, Y. (2024). Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy. arXiv preprint arXiv:2407.06813.
[6] Hofmann, V., Kalluri, P.R., Jurafsky, D. et al. AI generates covertly racist decisions about people based on their dialect. Nature 633, 147?154 (2024). https://doi.org/10.1038/s41586-024-07856-5
[7] Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A., & Lee, H. (2023). KoBBQ: Korean bias benchmark for question answering. arXiv preprint arXiv:2307.16778.
[8] Dong, Y. R., Hu, T., & Collier, N. (2024). Can LLM be a Personalized Judge?. arXiv preprint arXiv:2406.11657.
[9] Ashkinaze, J., Fry, E., Edara, N., Gilbert, E., & Budak, C. (2024). Plurals: A System for Guiding LLMs Via Simulated Social Ensembles. arXiv preprint arXiv:2409.17213.
[10] Feng, S., Sorensen, T., Liu, Y., Fisher, J., Park, C. Y., Choi, Y., & Tsvetkov, Y. (2024). Modular pluralism: Pluralistic alignment via multi-llm collaboration. arXiv preprint arXiv:2406.15951.
[11] “MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning” https://arxiv.org/abs/2310.16730
[12] “Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking” https://aclanthology.org/2024.findings-emnlp.879/