Body: <p>Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical rule? (2) a reduction of efficiency?</p>
  <p><strong>Update:</strong> By modification I mean the corresponding value after the action is taken. I am not referring to the Bellman update step. I am assuming we have filled our Q-table and now implementing in real scenario. And in this real scenario upon deciding on an action can I update the corresponding Q-value by say adding some constant factor? Not updating through Bellman eqn. Only after certain time steps I will re-train (if that's the correct word) and refresh my Q-table using the Bellman. I am thinking of like training and testing kind of phases here, if that make sense. I say so because my rewards are dependent on a random dynamic parameter and so I cannot just re-update my Q-table right after each time-step (to prevent unnecessary swaying of Q-values) and want to do so after certain <em>n</em> time-steps. In between, I just use the table to look up and take the actions without updating the corresponding Bellman update equation. But since I <em>have</em> &amp; <em>want</em> to do something to reflect some criteria upon taking the action, I am thinking what if I just modify the corresponding Q-values only by adding some constant factor until I retrain. This would help take better actions even until the <em>n</em> time-steps are over and ready for re-train....</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T06:21:01.540Z
Comments.Text: Could you clarify what you mean by modify: Changing the values stored in the table, or changing the values used to decide current action? What kinds of change are you considering - e.g. is it based on some mathematical property of the state, or some other collected data or random mutation? How often/by how much will you be changing the values? Without those details it is very hard to answer your questions (1) and (2).
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T12:56:47.553Z
Comments.Text: @NeilSlater I have added the required details. Please see it and give any suggestions.
Comments.UserDisplayName: knowledge_seeker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T15:05:28.733Z
Comments.Text: Thanks for the update. I don't understand why you would not want to use the Bellman update equation in your scenario? It is usually OK to use it in a live environment, and if you want to change the Q values, it would be the usual way to change them, whilst adding some constant does not seem to serve any purpose that I can think of. What are your concerns, and what do you think you will gain by adding this constant? You mention a Q-table a few times, but you have also tagged the question DQN, where there is a neural network instead of a Q-table - this may be an important detail.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T19:04:54.667Z
Comments.Text: @NeilSlater oh yes, I deleted the tag. I was arriving at a similar conclusion to what you said but I think I will first process what you said w.r.t my application. Will edit my code & then get back here to comment better on what you asked.
Comments.UserDisplayName: knowledge_seeker
Tags: <reinforcement-learning><machine-learning><q-learning><intelligent-agent>
Title: What if we modify some Q-values while taking the action?
UserDisplayName: knowledge_seeker
ThreadText: # What if we modify some Q-values while taking the action?
  
  (Asked by: username_0 on 2022-06-13)
  
  Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical rule? (2) a reduction of efficiency?
  
  **Update:** By modification I mean the corresponding value after the action is taken. I am not referring to the Bellman update step. I am assuming we have filled our Q-table and now implementing in real scenario. And in this real scenario upon deciding on an action can I update the corresponding Q-value by say adding some constant factor? Not updating through Bellman eqn. Only after certain time steps I will re-train (if that's the correct word) and refresh my Q-table using the Bellman. I am thinking of like training and testing kind of phases here, if that make sense. I say so because my rewards are dependent on a random dynamic parameter and so I cannot just re-update my Q-table right after each time-step (to prevent unnecessary swaying of Q-values) and want to do so after certain *n* time-steps. In between, I just use the table to look up and take the actions without updating the corresponding Bellman update equation. But since I *have* & *want* to do something to reflect some criteria upon taking the action, I am thinking what if I just modify the corresponding Q-values only by adding some constant factor until I retrain. This would help take better actions even until the *n* time-steps are over and ready for re-train....
  
  **Tags:** reinforcement-learning, machine-learning, q-learning, intelligent-agent
  > **Comments:**
  > **username_1**: Could you clarify what you mean by modify: Changing the values stored in the table, or changing the values used to decide current action? What kinds of change are you considering - e.g. is it based on some mathematical property of the state, or some other collected data or random mutation? How often/by how much will you be changing the values? Without those details it is very hard to answer your questions (1) and (2).  
  > **username_0**: @NeilSlater I have added the required details. Please see it and give any suggestions.  
  > **username_1**: Thanks for the update. I don't understand why you would not want to use the Bellman update equation in your scenario? It is usually OK to use it in a live environment, and if you want to change the Q values, it would be the usual way to change them, whilst adding some constant does not seem to serve any purpose that I can think of. What are your concerns, and what do you think you will gain by adding this constant? You mention a Q-table a few times, but you have also tagged the question DQN, where there is a neural network instead of a Q-table - this may be an important detail.  
  > **username_0**: @NeilSlater oh yes, I deleted the tag. I was arriving at a similar conclusion to what you said but I think I will first process what you said w.r.t my application. Will edit my code & then get back here to comment better on what you asked.  
  
  ---
  
  ## 0 answers
  
  