  In many real-world applications, the reward function is too complex to be
manually specified. In such cases, reward functions must instead be learned
from human feedback. Since the learned reward may fail to represent user
preferences, it is important to be able to validate the learned reward function
prior to deployment. One promising approach is to apply interpretability tools
to the reward function to spot potential deviations from the user's intention.
Existing work has applied general-purpose interpretability tools to understand
learned reward functions. We propose exploiting the intrinsic structure of
reward functions by first preprocessing them into simpler but equivalent reward
functions, which are then visualized. We introduce a general framework for such
reward preprocessing and propose concrete preprocessing algorithms. Our
empirical evaluation shows that preprocessed rewards are often significantly
easier to understand than the original reward.

Machine Learning