Asynchronous Methods for Deep Reinforcement Learning
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/DLR-RM/stable-baselines3pytorch★ 12,962
- github.com/toni-sm/skrljax★ 1,014
- github.com/tensorlayer/RLzootf★ 644
- github.com/Kaixhin/ACERpytorch★ 258
- github.com/lcswillems/torch-acpytorch★ 207
- github.com/nvlabs/gbrl_sb3pytorch★ 43
- github.com/ShibiHe/Q-Optimality-Tighteningnone★ 37
- github.com/brett-daley/fast-dqntf★ 8
- github.com/wtingda/DeepRLBreakouttf★ 6
- github.com/Zartris/TD3_continuous_controlpytorch★ 4
Abstract
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| Atari 2600 Alien | A3C FF hs | Score | 518.4 | — | Unverified |
| Atari 2600 Alien | A3C LSTM hs | Score | 945.3 | — | Unverified |
| Atari 2600 Alien | A3C FF (1 day) hs | Score | 182.1 | — | Unverified |
| Atari 2600 Amidar | A3C FF (1 day) hs | Score | 283.9 | — | Unverified |
| Atari 2600 Amidar | A3C FF hs | Score | 263.9 | — | Unverified |
| Atari 2600 Amidar | A3C LSTM hs | Score | 173 | — | Unverified |
| Atari 2600 Assault | A3C FF hs | Score | 5,474.9 | — | Unverified |
| Atari 2600 Assault | A3C LSTM hs | Score | 14,497.9 | — | Unverified |
| Atari 2600 Assault | A3C FF (1 day) hs | Score | 3,746.1 | — | Unverified |
| Atari 2600 Asterix | A3C LSTM hs | Score | 17,244.5 | — | Unverified |
| Atari 2600 Asterix | A3C FF hs | Score | 22,140.5 | — | Unverified |
| Atari 2600 Asterix | A3C FF (1 day) hs | Score | 6,723 | — | Unverified |
| Atari 2600 Asteroids | A3C FF (1 day) hs | Score | 3,009.4 | — | Unverified |
| Atari 2600 Asteroids | A3C FF hs | Score | 4,474.5 | — | Unverified |
| Atari 2600 Asteroids | A3C LSTM hs | Score | 5,093.1 | — | Unverified |
| Atari 2600 Atlantis | A3C FF (1 day) hs | Score | 772,392 | — | Unverified |
| Atari 2600 Atlantis | A3C FF hs | Score | 911,091 | — | Unverified |
| Atari 2600 Atlantis | A3C LSTM hs | Score | 875,822 | — | Unverified |
| Atari 2600 Bank Heist | A3C FF hs | Score | 970.1 | — | Unverified |
| Atari 2600 Bank Heist | A3C FF (1 day) hs | Score | 946 | — | Unverified |
| Atari 2600 Bank Heist | A3C LSTM hs | Score | 932.8 | — | Unverified |
| Atari 2600 Battle Zone | A3C LSTM hs | Score | 20,760 | — | Unverified |
| Atari 2600 Battle Zone | A3C FF (1 day) hs | Score | 11,340 | — | Unverified |
| Atari 2600 Battle Zone | A3C FF hs | Score | 12,950 | — | Unverified |
| Atari 2600 Beam Rider | A3C FF hs | Score | 22,707.9 | — | Unverified |
| Atari 2600 Beam Rider | A3C FF (1 day) hs | Score | 13,235.9 | — | Unverified |
| Atari 2600 Beam Rider | A3C LSTM hs | Score | 24,622.2 | — | Unverified |
| Atari 2600 Berzerk | A3C FF (1 day) hs | Score | 1,433.4 | — | Unverified |
| Atari 2600 Berzerk | A3C LSTM hs | Score | 862.2 | — | Unverified |
| Atari 2600 Berzerk | A3C FF hs | Score | 817.9 | — | Unverified |
| Atari 2600 Bowling | A3C FF hs | Score | 35.1 | — | Unverified |
| Atari 2600 Bowling | A3C LSTM hs | Score | 41.8 | — | Unverified |
| Atari 2600 Bowling | A3C FF (1 day) hs | Score | 36.2 | — | Unverified |
| Atari 2600 Boxing | A3C FF (1 day) hs | Score | 33.7 | — | Unverified |
| Atari 2600 Boxing | A3C FF hs | Score | 59.8 | — | Unverified |
| Atari 2600 Boxing | A3C LSTM hs | Score | 37.3 | — | Unverified |
| Atari 2600 Breakout | A3C FF (1 day) hs | Score | 551.6 | — | Unverified |
| Atari 2600 Breakout | A3C FF hs | Score | 681.9 | — | Unverified |
| Atari 2600 Breakout | A3C LSTM hs | Score | 766.8 | — | Unverified |
| Atari 2600 Centipede | A3C LSTM hs | Score | 1,997 | — | Unverified |
| Atari 2600 Centipede | A3C FF (1 day) hs | Score | 3,306.5 | — | Unverified |
| Atari 2600 Centipede | A3C FF hs | Score | 3,755.8 | — | Unverified |
| Atari 2600 Chopper Command | A3C FF (1 day) hs | Score | 4,669 | — | Unverified |
| Atari 2600 Chopper Command | A3C LSTM hs | Score | 10,150 | — | Unverified |
| Atari 2600 Chopper Command | A3C FF hs | Score | 7,021 | — | Unverified |
| Atari 2600 Crazy Climber | A3C FF (1 day) hs | Score | 101,624 | — | Unverified |
| Atari 2600 Crazy Climber | A3C LSTM hs | Score | 138,518 | — | Unverified |
| Atari 2600 Crazy Climber | A3C FF hs | Score | 112,646 | — | Unverified |
| Atari 2600 Demon Attack | A3C LSTM hs | Score | 115,201.9 | — | Unverified |
| Atari 2600 Demon Attack | A3C FF hs | Score | 113,308.4 | — | Unverified |