TD Learning with Constrained Gradients

Abstract

Temporal Difference Learning with function approximation is known to be unstable. Previous work like sutton2009fast and sutton2009convergent has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly mnih2015human. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird's counterexample keeps Q-learning from diverging.

Tasks

Reproductions