Answers.Body: <p>In addition to the RF [*], you also need to define an <em>exploratory policy</em> (an example is the <a href="https://ai.stackexchange.com/a/10495/2444"><span class="math-container">$\epsilon$</span>-greedy</a>), which allows you to explore the environment and learn the state-action value function <span class="math-container">$\hat{q}$</span>. Moreover, although you don't need to know the details (i.e. the specific probabilities of transitioning from one state to the other) of the <a href="https://ai.stackexchange.com/q/4456/2444">transition model</a>, often denoted by <span class="math-container">$p$</span>, you need a function that returns you the next state <span class="math-container">$s'$</span> for each action <span class="math-container">$a$</span> that you take in the current state <span class="math-container">$s$</span>. You may not need to define this function, but, for example, the next state could be given by some kind of simulator of the environment (for example, in the case of Atari games, the Atari simulator may provide you the next frame of the game, which you could use to build an approximation of the next state). You can read the Q-learning pseudocode <a href="https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html" rel="nofollow noreferrer">here</a>.</p>
  <p><sub>[*] The reward function is defined for the problem and, specifically, for the Markov Decision Process (MDP) that models the problem/environment. The RF is not defined only for applying the Q-learning to solve the problem (in fact, you could apply other algorithms, like SARSA), but you need the RF to use Q-learning; so, yes, you need to define/have the RF before applying Q-learning.  You can think of the RF as the learning signal that is used to guide the agent towards the optimal policy, and that's why it's specific to each environment/problem. Note that, in theory, there could be more than one RF that leads to the optimal policy for an environment (see <a href="https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf" rel="nofollow noreferrer">potential-based reward shaping</a> for more details). (This paragraph was addressing what I originally thought was the question: I'm leaving it here because it may be relevant to the readers).</sub></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-29T12:24:14.017Z
Answers.Text: A little bit of finessing may be useful. You don't necessarily need the full reward function in model-free agents. You need to be able to measure reward on each time step. In model-free learning, it is possible that there is a complex unknown reward function whilst the agent simply observes reward. However, it is also quite common that an environment does not have a built-in reward function and the agent must implement a function that converts observations into reward in order to fully define the problem.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-29T12:26:02.580Z
Answers.Text: @NeilSlater Yeah, you're right. You may not necessarily need to know the details of the reward function, as you may not also know the details of the transition model, but you need someone or something that provides you with the reward and states. I tried to state this for the case of the transition function.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T15:13:04.237Z
Answers.LastActivityDate: 2021-06-29T12:09:43.400Z
Answers.LastEditDate: 2021-06-29T12:09:43.400Z
Answers.UserDisplayName: nbro
Body: <p>In general, <span class="math-container">$Q$</span> function is defined as</p>
  <p><span class="math-container">$$Q : S \times A  \rightarrow \mathbb{R}$$</span>
  <span class="math-container">$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$</span></p>
  <p><span class="math-container">$\alpha$</span> and <span class="math-container">$\gamma$</span> are hyper-parameters. <span class="math-container">$r_{t+1}$</span> is the reward at next time step. <span class="math-container">$Q$</span> values are initialized arbitrarily.</p>
  <p>In addition to the reward function, which other functions do I need to implement Q-learning?</p>
  
Tags: <reinforcement-learning><q-learning><markov-decision-process><reward-functions><exploration-strategies>
Title: In addition to the reward function, which other functions do I need to implement Q-learning?
UserDisplayName: satya
ThreadText: # In addition to the reward function, which other functions do I need to implement Q-learning?
  
  (Asked by: username_0 on 2021-06-28)
  
  In general, $Q$ function is defined as
  
  $$Q : S \times A \rightarrow \mathbb{R}$$
  $$Q(s\_t,a\_t) = Q(s\_t,a\_t) + \alpha[r\_{t+1} + \gamma \max\limits\_{a} Q(s\_{t+1},a) - Q(s\_t,a\_t)] $$
  
  $\alpha$ and $\gamma$ are hyper-parameters. $r\_{t+1}$ is the reward at next time step. $Q$ values are initialized arbitrarily.
  
  In addition to the reward function, which other functions do I need to implement Q-learning?
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, reward-functions, exploration-strategies
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-28. Score: 2 | Accepted answer
  
  In addition to the RF [\*], you also need to define an *exploratory policy* (an example is the [$\epsilon$-greedy](https://ai.stackexchange.com/a/10495/2444)), which allows you to explore the environment and learn the state-action value function $\hat{q}$. Moreover, although you don't need to know the details (i.e. the specific probabilities of transitioning from one state to the other) of the [transition model](https://ai.stackexchange.com/q/4456/2444), often denoted by $p$, you need a function that returns you the next state $s'$ for each action $a$ that you take in the current state $s$. You may not need to define this function, but, for example, the next state could be given by some kind of simulator of the environment (for example, in the case of Atari games, the Atari simulator may provide you the next frame of the game, which you could use to build an approximation of the next state). You can read the Q-learning pseudocode [here](https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html).
  
  [\*] The reward function is defined for the problem and, specifically, for the Markov Decision Process (MDP) that models the problem/environment. The RF is not defined only for applying the Q-learning to solve the problem (in fact, you could apply other algorithms, like SARSA), but you need the RF to use Q-learning; so, yes, you need to define/have the RF before applying Q-learning. You can think of the RF as the learning signal that is used to guide the agent towards the optimal policy, and that's why it's specific to each environment/problem. Note that, in theory, there could be more than one RF that leads to the optimal policy for an environment (see [potential-based reward shaping](https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf) for more details). (This paragraph was addressing what I originally thought was the question: I'm leaving it here because it may be relevant to the readers).
  
  > **Comments:**
  > **username_2**: A little bit of finessing may be useful. You don't necessarily need the full reward function in model-free agents. You need to be able to measure reward on each time step. In model-free learning, it is possible that there is a complex unknown reward function whilst the agent simply observes reward. However, it is also quite common that an environment does not have a built-in reward function and the agent must implement a function that converts observations into reward in order to fully define the problem.  
  > **username_1**: @NeilSlater Yeah, you're right. You may not necessarily need to know the details of the reward function, as you may not also know the details of the transition model, but you need someone or something that provides you with the reward and states. I tried to state this for the case of the transition function.  
  
  ---
  
  