Exploratory Optimal Stopping: A Singular Control Formulation
<|reference_start|>Exploratory Optimal Stopping: A Singular Control Formulation: This paper explores continuous-time and state-space optimal stopping problems from a reinforcement learning perspective. We begin by formulating the stopping problem using randomized stopping times, where the decision maker's control is represented by the probability of stopping within a given time--specifically, a bounded, non-decreasing, c\`adl\`ag control process. To encourage exploration and facilitate learning, we introduce a regularized version of the problem by penalizing it with the cumulative residual entropy of the randomized stopping time. The regularized problem takes the form of an (n+1)-dimensional degenerate singular stochastic control with finite-fuel. We address this through the dynamic programming principle, which enables us to identify the unique optimal exploratory strategy. For the specific case of a real option problem, we derive a semi-explicit solution to the regularized problem, allowing us to assess the impact of entropy regularization and analyze the vanishing entropy limit. Finally, we propose a reinforcement learning algorithm based on policy iteration. We show both policy improvement and policy convergence results for our proposed algorithm.<|reference_end|>
@article{dianetti2024exploratory,
    title={Exploratory Optimal Stopping: A Singular Control Formulation},
    author={Jodi Dianetti, Giorgio Ferrari and Renyuan Xu},
    journal={arXiv preprint arXiv:2408.09335},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09335},
    primaryClass={math.OC cs.LG q-fin.MF stat.ML}
}