Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games

Code

• github.com/sundave1998/independent-npg-mpg
  OfficialIn paperpytorch★ 1

Abstract

This work studies an independent natural policy gradient (NPG) algorithm for the multi-agent reinforcement learning problem in Markov potential games. It is shown that, under mild technical assumptions and the introduction of the suboptimality gap, the independent NPG method with an oracle providing exact policy evaluation asymptotically reaches an -Nash Equilibrium (NE) within O(1/) iterations. This improves upon the previous best result of O(1/^2) iterations and is of the same order, O(1/), that is achievable for the single-agent case. Empirical results for a synthetic potential game and a congestion game are presented to verify the theoretical bounds.

Reproductions