SQT -- std Q-target

Abstract

Std Q-target is a conservative, actor-critic, ensemble, Q-learning-based algorithm, which is based on a single key Q-formula: Q-networks standard deviation, which is an "uncertainty penalty", and, serves as a minimalistic solution to the problem of overestimation bias. We implement SQT on top of TD3/TD7 code and test it against the state-of-the-art (SOTA) actor-critic algorithms, DDPG, TD3 and TD7 on seven popular MuJoCo and Bullet tasks. Our results demonstrate SQT's Q-target formula superiority over TD3's Q-target formula as a conservative solution to overestimation bias in RL, while SQT shows a clear performance advantage on a wide margin over DDPG, TD3, and TD7 on all tasks.

Tasks

Reproductions