Improving reinforcement learning algorithms: towards optimal learning rate policies

Code

• github.com/othmaneM/RL_adap_stepsize
  OfficialIn papernone★ 0

Abstract

This paper investigates to what extent one can improve reinforcement learning algorithms. Our study is split in three parts. First, our analysis shows that the classical asymptotic convergence rate O(1/N) is pessimistic and can be replaced by O(((N)/N)^) with 12 1 and N the number of iterations. Second, we propose a dynamic optimal policy for the choice of the learning rate (_k)_k 0 used in stochastic approximation (SA). We decompose our policy into two interacting levels: the inner and the outer level. In the inner level, we present the Alg:v_4_s algorithm (for "PAst Sign Search") which, based on a predefined sequence (^o_k)_k 0, constructs a new sequence (^i_k)_k 0 whose error decreases faster. In the outer level, we propose an optimal methodology for the selection of the predefined sequence (^o_k)_k 0. Third, we show empirically that our selection methodology of the learning rate outperforms significantly standard algorithms used in reinforcement learning (RL) in the three following applications: the estimation of a drift, the optimal placement of limit orders and the optimal execution of large number of shares.

Tasks

Reproductions