Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models
<|reference_start|>Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models: Learning a world model for model-free Reinforcement Learning (RL) agents can significantly improve the sample efficiency by learning policies in imagination. However, building a world model for Multi-Agent RL (MARL) can be particularly challenging due to the scalability issue in a centralized architecture arising from a large number of agents, and also the non-stationarity issue in a decentralized architecture stemming from the inter-dependency among agents. To address both challenges, we propose a novel world model for MARL that learns decentralized local dynamics for scalability, combined with a centralized representation aggregation from all agents. We cast the dynamics learning as an auto-regressive sequence modeling problem over discrete tokens by leveraging the expressive Transformer architecture, in order to model complex local dynamics across different agents and provide accurate and consistent long-term imaginations. As the first pioneering Transformer-based world model for multi-agent systems, we introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation within this context. Results on Starcraft Multi-Agent Challenge (SMAC) show that it outperforms strong model-free approaches and existing model-based methods in both sample efficiency and overall performance.<|reference_end|>
@article{zhang2024decentralized,
    title={Decentralized Transformers with Centralized Aggregation are
  Sample-Efficient Multi-Agent World Models},
    author={Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan, Xiu Li, Xuelong Li},
    journal={arXiv preprint arXiv:2406.15836},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.15836},
    primaryClass={cs.LG cs.AI cs.MA}
}