Answers.Body: <p>There's are some solutions to calculating q-values; find the exact values:</p>
  <ul>
  <li>Brute-force the action sequence to find max (not pratical)</li>
  <li>Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)</li>
  </ul>
  <p>Estimate the q-values:</p>
  <ul>
  <li>Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used</li>
  <li>Do a lot of randomisation to find max, including Monte Carlo tree search</li>
  </ul>
  <p>Gradually optimise the q-values:</p>
  <p>(1) Do optimisation based on Bellman equation (Q-learning):
  <span class="math-container">$$ q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$</span></p>
  <p>Bellman equation is true when the temporal difference (the part multiplied with <span class="math-container">$\alpha$</span>) reaches zero, which means the max of time t+1 reaches exact value.</p>
  <p>(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
  <span class="math-container">$$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T01:57:47.833Z
Answers.LastActivityDate: 2021-03-17T01:57:47.833Z
Answers.UserDisplayName: Dan D.
Body: <p>I need the q-value for my RL training, there are some approaches:</p>
  <ul>
  <li>Brute-force the action sequence (this won't work for long sequence)</li>
  <li>Use a classic algorithm to optimise and estimate (this ain't much AI)</li>
  <li>Create Monte Carlo samples and train an approximator network for calculating q-value</li>
  </ul>
  <p>I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-01T13:30:58.140Z
Comments.Text: How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-01T13:51:23.880Z
Comments.Text: anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo'
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-01T14:59:48.033Z
Comments.Text: Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-01T17:36:23.410Z
Comments.Text: @datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with David that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T02:04:59.993Z
Comments.Text: 'brute-force the action sequence' in future steps to get the best action at present state
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T02:06:11.093Z
Comments.Text: yeah, approaches to estimate q(s,a), to get the best (or at least good) action at current state
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T18:56:22.707Z
Comments.Text: @datdinhquoc Why don't you simply use Q-learning?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><monte-carlo-methods><value-functions>
Title: What are the popular approaches to estimating the Q-function?
UserDisplayName: Dan D.
ThreadText: # What are the popular approaches to estimating the Q-function?
  
  (Asked by: username_0 on 2021-02-01)
  
  I need the q-value for my RL training, there are some approaches:
  
  * Brute-force the action sequence (this won't work for long sequence)
  * Use a classic algorithm to optimise and estimate (this ain't much AI)
  * Create Monte Carlo samples and train an approximator network for calculating q-value
  
  I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?
  
  **Tags:** reinforcement-learning, q-learning, monte-carlo-methods, value-functions
  
  > **Comments:**
  > **username_1**: How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms?  
  > **username_0**: anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo'  
  > **username_1**: Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question.  
  > **username_2**: @datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with username_1 that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)?  
  > **username_0**: 'brute-force the action sequence' in future steps to get the best action at present state  
  > **username_0**: yeah, approaches to estimate q(s,a), to get the best (or at least good) action at current state  
  > **username_2**: @datdinhquoc Why don't you simply use Q-learning?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-03-17. Score: 0 | Accepted answer
  
  There's are some solutions to calculating q-values; find the exact values:
  
  * Brute-force the action sequence to find max (not pratical)
  * Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)
  
  Estimate the q-values:
  
  * Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
  * Do a lot of randomisation to find max, including Monte Carlo tree search
  
  Gradually optimise the q-values:
  
  (1) Do optimisation based on Bellman equation (Q-learning):
  $$ q\_t(s\_t,a\_t) = q\_t(s\_t,a\_t) + \alpha(r + \gamma\times\max(q\_{t+1}(s\_{t+1},a\_{t+1})) - q\_t(s\_t,a\_t))$$
  
  Bellman equation is true when the temporal difference (the part multiplied with $\alpha$) reaches zero, which means the max of time t+1 reaches exact value.
  
  (2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
  $$r + \gamma\times\max(q\_{t+1}(s\_{t+1},a\_{t+1}))$$
  
  
  ---
  
  