  The ability to learn good representations of states is essential for solving
large reinforcement learning problems, where exploration, generalization, and
transfer are particularly challenging. The Laplacian representation is a
promising approach to address these problems by inducing intrinsic rewards for
temporally-extended action discovery and reward shaping, and informative state
encoding. To obtain the Laplacian representation one needs to compute the
eigensystem of the graph Laplacian, which is often approximated through
optimization objectives compatible with deep learning approaches. These
approximations, however, depend on hyperparameters that are impossible to tune
efficiently, converge to arbitrary rotations of the desired eigenvectors, and
are unable to accurately recover the corresponding eigenvalues. In this paper
we introduce a theoretically sound objective and corresponding optimization
algorithm for approximating the Laplacian representation. Our approach
naturally recovers both the true eigenvectors and eigenvalues while eliminating
the hyperparameter dependence of previous approximations. We provide
theoretical guarantees for our method and we show that those results translate
empirically into robust learning across multiple environments.

Machine Learning;Artificial Intelligence