QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/oxwhirl/pymarlOfficialpytorch★ 2,168
- github.com/facebookresearch/benchmarlpytorch★ 588
- github.com/hhhusiyi-monash/UPDeTpytorch★ 137
- github.com/nju-rl/acormpytorch★ 63
- github.com/TonghanWang/DOPpytorch★ 51
- github.com/puyuan1996/MARLpytorch★ 41
- github.com/cathyhxh/ctdspytorch★ 12
- github.com/ifpen/wfcrl-benchmarkpytorch★ 9
- github.com/jugg1er/airpytorch★ 8
- github.com/gingkg/smacpytorch★ 0
Abstract
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.