  Visual imitation learning enables reinforcement learning agents to learn to
behave from expert visual demonstrations such as videos or image sequences,
without explicit, well-defined rewards. Previous research either adopted
supervised learning techniques or induce simple and coarse scalar rewards from
pixels, neglecting the dense information contained in the image demonstrations.
In this work, we propose to measure the expertise of various local regions of
image samples, or called \textit{patches}, and recover multi-dimensional
\textit{patch rewards} accordingly. Patch reward is a more precise rewarding
characterization that serves as a fine-grained expertise measurement and visual
explainability tool. Specifically, we present Adversarial Imitation Learning
with Patch Rewards (PatchAIL), which employs a patch-based discriminator to
measure the expertise of different local parts from given images and provide
patch rewards. The patch-based knowledge is also used to regularize the
aggregated reward and stabilize the training. We evaluate our method on
DeepMind Control Suite and Atari tasks. The experiment results have
demonstrated that PatchAIL outperforms baseline methods and provides valuable
interpretations for visual demonstrations.

Artificial Intelligence