Value function interference and greedy action selection in value-based multi-objective reinforcement learning
<|reference_start|>Value function interference and greedy action selection in value-based multi-objective reinforcement learning: Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) to the more general case of problems with multiple, conflicting objectives, represented by vector-valued rewards. Widely-used scalar RL methods such as Q-learning can be modified to handle multiple objectives by (1) learning vector-valued value functions, and (2) performing action selection using a scalarisation or ordering operator which reflects the user's utility with respect to the different objectives. However, as we demonstrate here, if the user's utility function maps widely varying vector-values to similar levels of utility, this can lead to interference in the value-function learned by the agent, leading to convergence to sub-optimal policies. This will be most prevalent in stochastic environments when optimising for the Expected Scalarised Return criterion, but we present a simple example showing that interference can also arise in deterministic environments. We demonstrate empirically that avoiding the use of random tie-breaking when identifying greedy actions can ameliorate, but not fully overcome, the problems caused by value function interference.<|reference_end|>
@article{vamplew2024value,
    title={Value function interference and greedy action selection in value-based
  multi-objective reinforcement learning},
    author={Peter Vamplew, Cameron Foale, Richard Dazeley},
    journal={arXiv preprint arXiv:2402.06266},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.06266},
    primaryClass={cs.LG}
}