The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure

Abstract

Many reinforcement learning (RL) algorithms are too costly to use in practice due to the large sizes S, A of the problem's state and action space. To resolve this issue, we study transfer RL with latent low rank structure. We consider the problem of transferring a latent low rank representation when the source and target MDPs have transition kernels with Tucker rank (S , d, A ), (S , S , d), (d, S, A ), or (d , d , d ). In each setting, we introduce the transfer-ability coefficient that measures the difficulty of representational transfer. Our algorithm learns latent representations in each source MDP and then exploits the linear structure to remove the dependence on S, A , or S A in the target MDP regret bound. We complement our positive results with information theoretic lower bounds that show our algorithms (excluding the (d, d, d) setting) are minimax-optimal with respect to .

Tasks

Reproductions