  Learning from Preferential Feedback (LfPF) plays an essential role in
training Large Language Models, as well as certain types of interactive
learning agents. However, a substantial gap exists between the theory and
application of LfPF algorithms. Current results guaranteeing the existence of
optimal policies in LfPF problems assume that both the preferences and
transition dynamics are determined by a Markov Decision Process. We introduce
the Direct Preference Process, a new framework for analyzing LfPF problems in
partially-observable, non-Markovian environments. Within this framework, we
establish conditions that guarantee the existence of optimal policies by
considering the ordinal structure of the preferences. Using the von
Neumann-Morgenstern Expected Utility Theorem, we show that the Direct
Preference Process generalizes the standard reinforcement learning problem. Our
findings narrow the gap between the empirical success and theoretical
understanding of LfPF algorithms and provide future practitioners with the
tools necessary for a more principled design of LfPF agents.

Machine Learning