<p>There's are some solutions to calculating q-values; find the exact values:</p>
<ul>
<li>Brute-force the action sequence to find max (not pratical)</li>
<li>Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)</li>
</ul>
<p>Estimate the q-values:</p>
<ul>
<li>Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used</li>
<li>Do a lot of randomisation to find max, including Monte Carlo tree search</li>
</ul>
<p>Gradually optimise the q-values:</p>
<p>(1) Do optimisation based on Bellman equation (Q-learning):
<span class="math-container">$$ q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$</span></p>
<p>Bellman equation is true when the temporal difference (the part multiplied with <span class="math-container">$\alpha$</span>) reaches zero, which means the max of time t+1 reaches exact value.</p>
<p>(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
<span class="math-container">$$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$</span></p>

CC BY-SA 4.0
2021-03-17T01:57:47.833Z
2021-03-17T01:57:47.833Z
Dan D.
<p>I need the q-value for my RL training, there are some approaches:</p>
<ul>
<li>Brute-force the action sequence (this won't work for long sequence)</li>
<li>Use a classic algorithm to optimise and estimate (this ain't much AI)</li>
<li>Create Monte Carlo samples and train an approximator network for calculating q-value</li>
</ul>
<p>I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?</p>

CC BY-SA 4.0
2021-02-01T13:30:58.140Z
How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms?
David
CC BY-SA 4.0
2021-02-01T13:51:23.880Z
anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo'
Dan D.
CC BY-SA 4.0
2021-02-01T14:59:48.033Z
Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question.
David
CC BY-SA 4.0
2021-02-01T17:36:23.410Z
@datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with David that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)?
nbro
CC BY-SA 4.0
2021-02-02T02:04:59.993Z
'brute-force the action sequence' in future steps to get the best action at present state
Dan D.
CC BY-SA 4.0
2021-02-02T02:06:11.093Z
yeah, approaches to estimate q(s,a), to get the best (or at least good) action at current state
Dan D.
CC BY-SA 4.0
2021-02-02T18:56:22.707Z
@datdinhquoc Why don't you simply use Q-learning?
nbro
<reinforcement-learning><q-learning><monte-carlo-methods><value-functions>
What are the popular approaches to estimating the Q-function?
Dan D.
# What are the popular approaches to estimating the Q-function?

(Asked by: username_0 on 2021-02-01)

I need the q-value for my RL training, there are some approaches:

* Brute-force the action sequence (this won't work for long sequence)
* Use a classic algorithm to optimise and estimate (this ain't much AI)
* Create Monte Carlo samples and train an approximator network for calculating q-value

I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?

**Tags:** reinforcement-learning, q-learning, monte-carlo-methods, value-functions

> **Comments:**
> **username_1**: How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms?  
> **username_0**: anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo'  
> **username_1**: Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question.  
> **username_2**: @datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with username_1 that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)?  
> **username_0**: 'brute-force the action sequence' in future steps to get the best action at present state  
> **username_0**: yeah, approaches to estimate q(s,a), to get the best (or at least good) action at current state  
> **username_2**: @datdinhquoc Why don't you simply use Q-learning?  

---

## 1 answers

### Answer by username_0 on 2021-03-17. Score: 0 | Accepted answer

There's are some solutions to calculating q-values; find the exact values:

* Brute-force the action sequence to find max (not pratical)
* Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)

Estimate the q-values:

* Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
* Do a lot of randomisation to find max, including Monte Carlo tree search

Gradually optimise the q-values:

(1) Do optimisation based on Bellman equation (Q-learning):
$$ q\_t(s\_t,a\_t) = q\_t(s\_t,a\_t) + \alpha(r + \gamma\times\max(q\_{t+1}(s\_{t+1},a\_{t+1})) - q\_t(s\_t,a\_t))$$

Bellman equation is true when the temporal difference (the part multiplied with $\alpha$) reaches zero, which means the max of time t+1 reaches exact value.

(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
$$r + \gamma\times\max(q\_{t+1}(s\_{t+1},a\_{t+1}))$$


---

