ICML Conference (2025)
Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Kanghoon Lee, Woohyung Lim
Abstract
Multi-Agent Reinforcement Learning (MARL) faces challenges in sparse reward environments, where agents struggle to learn effective strategies. Macro-actions—sequences of actions executed as single decisions—facilitate long-term planning but introduce asynchrony due to varying durations across agents, complicating Centralized Training with Decentralized Execution (CTDE). Existing methods handle this with padding, which misaligns asynchronous experiences, leading to spurious correlations.We propose an Agent-Centric Actor-Critic (ACAC) algorithm that addresses asynchrony without relying on padding. ACAC employs agent-centric history encoders for independent trajectory processing and an attention-based centralized critic to integrate agent information.Experiments on macro-action benchmarks show that ACAC accelerates convergence and enhances overall performance compared to baseline methods, demonstrating its effectiveness in complex MARL tasks.