A2C is a special case of PPO

2022-05-18Code Available1· sign in to hype

Shengyi Huang, Anssi Kanervisto, Antonin Raffin, Weixun Wang, Santiago Ontañón, Rousslan Fernand Julien Dossa

Code Available — Be the first to reproduce this paper.

Code

github.com/vwxyzjn/a2c_is_a_special_case_of_ppo
OfficialIn paperpytorch★ 22

Abstract

Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using Stable-baselines3, showing A2C and PPO produce the exact same models when other settings are controlled.

Tasks

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

A2C is a special case of PPO

Code

Abstract

Tasks

Reproductions