Answers.Body: <blockquote>
  <p>Is my definition of 'state' or 'action' wrong?</p>
  </blockquote>
  <p>I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand.</p>
  <p>In RL theory, the set of all nodes and the links between them is called the environment, the agent's current location is the state, and taking a route from one node to the next is the action. The reward is just as you said, except that usually the agent is trying to maximize the reward.</p>
  <p>Once your terms are aligned, regular Q learning (which keeps a simple table of Q values instead of approximating the Q values with something like a neural network) should actually work better than deep Q learning on such a small problem, because training a neural net can be challenging.</p>
  <p>Which brings us to the changing rewards. When the rewards change, the agent will need to find the new optimal path. To do that it will need to explore more than it would for a problem with fixed rewards that can settle into an optimal solution. For example, if you were using epsilon greedy exploration, epsilon would need to remain large, or be reset to a large value when the rewards change.</p>
  <p>Of course, the agent will perform poorly until it adapts to the new rewards, but you might be able to recalculate the optimal route when the parameters change by running the simulation for some time before making any real decisions.</p>
  <p>Hope that helps</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:16:56.613Z
Answers.Text: Thanks for the detailed answer. I understand your view about dealing with changing rewards part. Actually this was precisely the reason why I was thinking that the tabular approach might not work. The below comment was my thought process. Please read it and tell me if there is a flaw.
Answers.UserDisplayName: knowledge_seeker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:21:30.060Z
Answers.Text: My thought process, "ok, if I take state as the current position (which is a node) it would be finite in number. But as much as I may define the state as the current position i.e. a node number, my reward still depends on a link metric that can take any value which is similar to choosing a link as a state since choosing which next link or which next node are synonymous. And so maybe I need a DQN as the link metric will take different values".  I think I confuse between choosing a link vs node number i.e. current position as my state...
Answers.UserDisplayName: knowledge_seeker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:37:40.933Z
Answers.Text: "choosing which next link or which next node are synonymous". Yep, you could think of that either way. "And so maybe I need a DQN as the link metric will take different values" I don't follow this part. A Q value in the table of Q values in regular Q learning can also change, and will change faster and be more accurate than an estimated Q value from the neural network in DQN.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:40:41.743Z
Answers.Text: Perhaps this will help: Q learning calculates a function Q(s,a) whose values are precisely updated according to the Bellman equation. DQN tries to estimate Q(s,a), but those estimates aren't as good as knowing the true value.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:43:05.040Z
Answers.Text: I understand. And what I meant by that "And so.." was, if I consider the selecting next link as action, link is something whose metrics change and those changes are not finite i.e. can take any value. So estimating Q-value would be like estimating for a continuous state space.......But with the idea you gave towards the end of your answer, I think I can do that with a tabular one by maybe avoiding updating the Q-table upon every single successful node or route discovery completion?
Answers.UserDisplayName: knowledge_seeker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T02:45:57.323Z
Answers.Text: It's ok if the metrics/rewards and the Q-values are continuous, only the number of actions (options) needs to be finite. For example, if there were two links leading out of a node, the Q value table would only need two entries for that node, one per link, even if the metric/reward for each link were unconstrained.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T18:12:53.253Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/136973/discussion-between-knowledge-seeker-and-lee-reeves).
Answers.UserDisplayName: knowledge_seeker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T18:18:27.843Z
Answers.Text: Ok, I'm there now
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-09T23:21:24.133Z
Answers.LastActivityDate: 2022-06-09T23:45:59.750Z
Answers.LastEditDate: 2022-06-09T23:45:59.750Z
Answers.UserDisplayName: Lee Reeves
Body: <p>I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried adapting the code from the article for my scenario. Visiting this page is not required, just gave the link for reference. <a href="https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/" rel="nofollow noreferrer">https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/</a></p>
  <p><strong>My Scenario:</strong> As an analogy to my network routing, say my source &amp; destination are node A and node E. Say only two routes are possible: A-B-E and A-B-C-E. I want to select the next node at each 'action' stage that finally together forms a route. Hence, I take my 'state' and 'action' both as the set of all nodes.</p>
  <p><strong>Problem:</strong> BUT my reward is a value I assign that depends on the parameters of the links connecting the nodes. i.e.if I compute some network metric as x for the link A-B then I give this x as the reward since I want to minimize the x in my network. And obviously, this x would differ over time since my network links' status changes. It is not something I can pre-define for a given state-action as in the article example like obstacle means negative reward, destination means positive reward,etc.</p>
  <p>So what should I do in this case? Is my definition of 'state' or 'action' wrong? Or do I need a deep Q-network? Is it for such kinds of cases? Please suggest.</p>
  
Tags: <reinforcement-learning><machine-learning><deep-rl><q-learning><dqn>
Title: How to deal with changing rewards in Q-learning? DQN?
UserDisplayName: knowledge_seeker
ThreadText: # How to deal with changing rewards in Q-learning? DQN?
  
  (Asked by: username_0 on 2022-06-09)
  
  I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried adapting the code from the article for my scenario. Visiting this page is not required, just gave the link for reference. <https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/>
  
  **My Scenario:** As an analogy to my network routing, say my source & destination are node A and node E. Say only two routes are possible: A-B-E and A-B-C-E. I want to select the next node at each 'action' stage that finally together forms a route. Hence, I take my 'state' and 'action' both as the set of all nodes.
  
  **Problem:** BUT my reward is a value I assign that depends on the parameters of the links connecting the nodes. i.e.if I compute some network metric as x for the link A-B then I give this x as the reward since I want to minimize the x in my network. And obviously, this x would differ over time since my network links' status changes. It is not something I can pre-define for a given state-action as in the article example like obstacle means negative reward, destination means positive reward,etc.
  
  So what should I do in this case? Is my definition of 'state' or 'action' wrong? Or do I need a deep Q-network? Is it for such kinds of cases? Please suggest.
  
  **Tags:** reinforcement-learning, machine-learning, deep-rl, q-learning, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-09. Score: 1 | Accepted answer
  
  > Is my definition of 'state' or 'action' wrong?
  
  I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand.
  
  In RL theory, the set of all nodes and the links between them is called the environment, the agent's current location is the state, and taking a route from one node to the next is the action. The reward is just as you said, except that usually the agent is trying to maximize the reward.
  
  Once your terms are aligned, regular Q learning (which keeps a simple table of Q values instead of approximating the Q values with something like a neural network) should actually work better than deep Q learning on such a small problem, because training a neural net can be challenging.
  
  Which brings us to the changing rewards. When the rewards change, the agent will need to find the new optimal path. To do that it will need to explore more than it would for a problem with fixed rewards that can settle into an optimal solution. For example, if you were using epsilon greedy exploration, epsilon would need to remain large, or be reset to a large value when the rewards change.
  
  Of course, the agent will perform poorly until it adapts to the new rewards, but you might be able to recalculate the optimal route when the parameters change by running the simulation for some time before making any real decisions.
  
  Hope that helps
  
  > **Comments:**
  > **username_0**: Thanks for the detailed answer. I understand your view about dealing with changing rewards part. Actually this was precisely the reason why I was thinking that the tabular approach might not work. The below comment was my thought process. Please read it and tell me if there is a flaw.  
  > **username_0**: My thought process, "ok, if I take state as the current position (which is a node) it would be finite in number. But as much as I may define the state as the current position i.e. a node number, my reward still depends on a link metric that can take any value which is similar to choosing a link as a state since choosing which next link or which next node are synonymous. And so maybe I need a DQN as the link metric will take different values".  I think I confuse between choosing a link vs node number i.e. current position as my state...  
  > **username_1**: "choosing which next link or which next node are synonymous". Yep, you could think of that either way. "And so maybe I need a DQN as the link metric will take different values" I don't follow this part. A Q value in the table of Q values in regular Q learning can also change, and will change faster and be more accurate than an estimated Q value from the neural network in DQN.  
  > **username_1**: Perhaps this will help: Q learning calculates a function Q(s,a) whose values are precisely updated according to the Bellman equation. DQN tries to estimate Q(s,a), but those estimates aren't as good as knowing the true value.  
  > **username_0**: I understand. And what I meant by that "And so.." was, if I consider the selecting next link as action, link is something whose metrics change and those changes are not finite i.e. can take any value. So estimating Q-value would be like estimating for a continuous state space.......But with the idea you gave towards the end of your answer, I think I can do that with a tabular one by maybe avoiding updating the Q-table upon every single successful node or route discovery completion?  
  > **username_1**: It's ok if the metrics/rewards and the Q-values are continuous, only the number of actions (options) needs to be finite. For example, if there were two links leading out of a node, the Q value table would only need two entries for that node, one per link, even if the metric/reward for each link were unconstrained.  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/136973/discussion-between-knowledge-seeker-and-lee-reeves).  
  > **username_1**: Ok, I'm there now  
  
  ---
  
  