Body: <p>The title may seem too broad, let me specify this question a bit more.</p>
  <p>Suppose that there is a problem that can be solved via MCMC-based algorithm, i.e., for its formulation we can construct a Markov chain, and Monte Carlo sampling is adopted to reach a desired stationary distribution.</p>
  <p>If this MCMC algorithm has proved to work well, then is it possible that its Markov chain-based description of the problem can also be used to formulate an RL description?</p>
  <p>If so, there is another more detailed question. Assuming that the acceptance probability in MCMC algorithm is calculated based on Metropolis-Hastings, say</p>
  <p><span class="math-container">$$
  A(x^{\prime}, x)=\min (1, \frac{P(x^{\prime})}{P(x)} \frac{g(x\mid x^{\prime})}{g(x^{\prime}\mid x)}).
  $$</span></p>
  <p>What could the relationship of the transition model of an RL model and this acceptance probability be?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-07T15:47:04.993Z
Comments.Text: I don't really see much common ground between the two... one is used to sample a complex distribution, the other one traverse a distribution (state space) and either is used to learn a policy that optimize a reward, or in other cases just to estimate a gradient...
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-07T16:35:38.247Z
Comments.Text: I agree that the two don't have much in common, but I just want to find out whether an existing MCMC algorithm could be regarded as a reference for formulating the problem as MDP, which RL model is usually based on.
Comments.UserDisplayName: Halve Luve
Tags: <reinforcement-learning><machine-learning><markov-chain-monte-carlo>
Title: What's the relationship between Reinforcement Learning (RL) and Markov Chain Monte Carlo (MCMC)?
UserDisplayName: Halve Luve
ThreadText: # What's the relationship between Reinforcement Learning (RL) and Markov Chain Monte Carlo (MCMC)?
  
  (Asked by: username_0 on 2023-09-07)
  
  The title may seem too broad, let me specify this question a bit more.
  
  Suppose that there is a problem that can be solved via MCMC-based algorithm, i.e., for its formulation we can construct a Markov chain, and Monte Carlo sampling is adopted to reach a desired stationary distribution.
  
  If this MCMC algorithm has proved to work well, then is it possible that its Markov chain-based description of the problem can also be used to formulate an RL description?
  
  If so, there is another more detailed question. Assuming that the acceptance probability in MCMC algorithm is calculated based on Metropolis-Hastings, say
  
  $$
  A(x^{\prime}, x)=\min (1, \frac{P(x^{\prime})}{P(x)} \frac{g(x\mid x^{\prime})}{g(x^{\prime}\mid x)}).
  $$
  
  What could the relationship of the transition model of an RL model and this acceptance probability be?
  
  **Tags:** reinforcement-learning, machine-learning, markov-chain-monte-carlo
  > **Comments:**
  > **username_1**: I don't really see much common ground between the two... one is used to sample a complex distribution, the other one traverse a distribution (state space) and either is used to learn a policy that optimize a reward, or in other cases just to estimate a gradient...  
  > **username_0**: I agree that the two don't have much in common, but I just want to find out whether an existing MCMC algorithm could be regarded as a reference for formulating the problem as MDP, which RL model is usually based on.  
  
  ---
  
  ## 0 answers
  
  