Skill Discovery for Exploration and Planning using Deep Skill Graphs

Abstract

Temporal abstraction provides an opportunity to drastically lower the decision making burden facing reinforcement learning agents in rich sensorimotor spaces. Well constructed hierarchies induce state and action abstractions that can reduce large continuous MDPs into small discrete ones, in which planning with a learned model is feasible. We propose a novel algorithm, Deep Skill Graphs, for acquiring such a minimal representation of an environment. Our algorithm seamlessly interleaves discovering skills and planning using them to gain unsupervised mastery over ever increasing portions of the state-space. The constructed skill graph can be used to drive the agent to novel goals at test time, requiring little-to-no additional learning. We test our algorithm on a series of continuous control tasks where it outperforms baseline flat and hierarchical RL methods alike.

Tasks

Reproductions