On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems
<|reference_start|>On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems: In Reinforcement Learning-based Recommender Systems (RLRS), the complexity and dynamism of user interactions often result in high-dimensional and noisy state spaces, making it challenging to discern which aspects of the state are truly influential in driving the decision-making process. This issue is exacerbated by the evolving nature of user preferences and behaviors, requiring the recommender system to adaptively focus on the most relevant information for decision-making while preserving generaliability. To tackle this problem, we introduce an innovative causal approach for decomposing the state and extracting \textbf{C}ausal-\textbf{I}n\textbf{D}ispensable \textbf{S}tate Representations (CIDS) in RLRS. Our method concentrates on identifying the \textbf{D}irectly \textbf{A}ction-\textbf{I}nfluenced \textbf{S}tate Variables (DAIS) and \textbf{A}ction-\textbf{I}nfluence \textbf{A}ncestors (AIA), which are essential for making effective recommendations. By leveraging conditional mutual information, we develop a framework that not only discerns the causal relationships within the generative process but also isolates critical state variables from the typically dense and high-dimensional state representations. We provide theoretical evidence for the identifiability of these variables. Then, by making use of the identified causal relationship, we construct causal-indispensable state representations, enabling the training of policies over a more advantageous subset of the agent's state space. We demonstrate the efficacy of our approach through extensive experiments, showcasing our method outperforms state-of-the-art methods.<|reference_end|>
@article{wang2024on,
    title={On Causally Disentangled State Representation Learning for Reinforcement
  Learning based Recommender Systems},
    author={Siyu Wang and Xiaocong Chen and Lina Yao},
    journal={arXiv preprint arXiv:2407.13091},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13091},
    primaryClass={cs.AI cs.IR}
}