Learning values across many orders of magnitude
Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, David Silver
Unverified — Be the first to reproduce this paper.
ReproduceAbstract
Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were all clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using the adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| Atari 2600 Alien | DDQN+Pop-Art noop | Score | 3,213.5 | — | Unverified |
| Atari 2600 Amidar | DDQN+Pop-Art noop | Score | 782.5 | — | Unverified |
| Atari 2600 Assault | DDQN+Pop-Art noop | Score | 9,011.6 | — | Unverified |
| Atari 2600 Asterix | DDQN+Pop-Art noop | Score | 18,919.5 | — | Unverified |
| Atari 2600 Asteroids | DDQN+Pop-Art noop | Score | 2,869.3 | — | Unverified |
| Atari 2600 Atlantis | DDQN+Pop-Art noop | Score | 340,076 | — | Unverified |
| Atari 2600 Bank Heist | DDQN+Pop-Art noop | Score | 1,103.3 | — | Unverified |
| Atari 2600 Battle Zone | DDQN+Pop-Art noop | Score | 8,220 | — | Unverified |
| Atari 2600 Beam Rider | DDQN+Pop-Art noop | Score | 8,299.4 | — | Unverified |
| Atari 2600 Berzerk | DDQN+Pop-Art noop | Score | 1,199.6 | — | Unverified |
| Atari 2600 Bowling | DDQN+Pop-Art noop | Score | 102.1 | — | Unverified |
| Atari 2600 Boxing | DDQN+Pop-Art noop | Score | 99.3 | — | Unverified |
| Atari 2600 Breakout | DDQN+Pop-Art noop | Score | 344.1 | — | Unverified |
| Atari 2600 Centipede | DDQN+Pop-Art noop | Score | 49,065.8 | — | Unverified |
| Atari 2600 Chopper Command | DDQN+Pop-Art noop | Score | 775 | — | Unverified |
| Atari 2600 Crazy Climber | DDQN+Pop-Art noop | Score | 119,679 | — | Unverified |
| Atari 2600 Demon Attack | DDQN+Pop-Art noop | Score | 63,644.9 | — | Unverified |
| Atari 2600 Double Dunk | DDQN+Pop-Art noop | Score | -11.5 | — | Unverified |
| Atari 2600 Enduro | DDQN+Pop-Art noop | Score | 2,002.1 | — | Unverified |
| Atari 2600 Fishing Derby | DDQN+Pop-Art noop | Score | 45.1 | — | Unverified |
| Atari 2600 Freeway | DDQN+Pop-Art noop | Score | 33.4 | — | Unverified |
| Atari 2600 Frostbite | DDQN+Pop-Art noop | Score | 3,469.6 | — | Unverified |
| Atari 2600 Gopher | DDQN+Pop-Art noop | Score | 56,218.2 | — | Unverified |
| Atari 2600 Gravitar | DDQN+Pop-Art noop | Score | 483.5 | — | Unverified |
| Atari 2600 HERO | DDQN+Pop-Art noop | Score | 14,225.2 | — | Unverified |
| Atari 2600 Ice Hockey | DDQN+Pop-Art noop | Score | -4.1 | — | Unverified |
| Atari 2600 James Bond | DDQN+Pop-Art noop | Score | 507.5 | — | Unverified |
| Atari 2600 Kangaroo | DDQN+Pop-Art noop | Score | 13,150 | — | Unverified |
| Atari 2600 Krull | DDQN+Pop-Art noop | Score | 9,745.1 | — | Unverified |
| Atari 2600 Kung-Fu Master | DDQN+Pop-Art noop | Score | 34,393 | — | Unverified |
| Atari 2600 Ms. Pacman | DDQN+Pop-Art noop | Score | 4,963.8 | — | Unverified |
| Atari 2600 Name This Game | DDQN+Pop-Art noop | Score | 15,851.2 | — | Unverified |
| Atari 2600 Pong | DDQN+Pop-Art noop | Score | 20.6 | — | Unverified |
| Atari 2600 Private Eye | DDQN+Pop-Art noop | Score | 286.7 | — | Unverified |
| Atari 2600 Q*Bert | DDQN+Pop-Art noop | Score | 5,236.8 | — | Unverified |
| Atari 2600 River Raid | DDQN+Pop-Art noop | Score | 12,530.8 | — | Unverified |
| Atari 2600 Road Runner | DDQN+Pop-Art noop | Score | 47,770 | — | Unverified |
| Atari 2600 Robotank | DDQN+Pop-Art noop | Score | 64.3 | — | Unverified |
| Atari 2600 Seaquest | DDQN+Pop-Art noop | Score | 10,932.3 | — | Unverified |
| Atari 2600 Space Invaders | DDQN+Pop-Art noop | Score | 2,589.7 | — | Unverified |
| Atari 2600 Star Gunner | DDQN+Pop-Art noop | Score | 589 | — | Unverified |
| Atari 2600 Tennis | DDQN+Pop-Art noop | Score | 12.1 | — | Unverified |
| Atari 2600 Time Pilot | DDQN+Pop-Art noop | Score | 4,870 | — | Unverified |
| Atari 2600 Tutankham | DDQN+Pop-Art noop | Score | 183.9 | — | Unverified |
| Atari 2600 Up and Down | DDQN+Pop-Art noop | Score | 22,474.4 | — | Unverified |
| Atari 2600 Venture | DDQN+Pop-Art noop | Score | 1,172 | — | Unverified |
| Atari 2600 Video Pinball | DDQN+Pop-Art noop | Score | 56,287 | — | Unverified |
| Atari 2600 Wizard of Wor | DDQN+Pop-Art noop | Score | 483 | — | Unverified |
| Atari 2600 Zaxxon | DDQN+Pop-Art noop | Score | 14,402 | — | Unverified |