Body: <p>I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action <span class="math-container">$a$</span> can change. Even the reward for the same <span class="math-container">$(s, a)$</span> pair can change with time too.</p>
  <p>In my problem, a UAV flies and visits several devices that would go through non-stationary environments. Each visit of the UAV can result in a performance improvement for the visited device. This happens gradually until a new environment appears for this device.</p>
  <p>So, the state is the time elapsed since the last time it visited each device. The action is the chosen device. The reward is the summation of the average cost of <strong>the current visited device and estimated rewards of other unvisited devices</strong>. This results in the fact that the UAV's reward can change. Most of the time, it increases gradually. Occasionally, it drops because of a new environment. Like a staircase going up, up, up, and suddenly down. Then it goes up and repeats. The time frequency of a new environment for each device is unique.</p>
  <p>I tried the above state and reward settings. When I use Actor-Critic to solve it and only use the reward of the current visited device, the UAV tends to visit one device only and ignores other devices. But when I add the reward of other unvisited devices, no convergence can be seen. In this case, the UAV tends to visit only one device occasionally, even when the reward is extremely low.</p>
  <p>My questions are:</p>
  <ol>
  <li><p>Am I setting the state and reward in the right way?</p>
  </li>
  <li><p>What algorithm should I use here?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T12:42:49.683Z
Comments.Text: Sounds like it settles down too early.  It needs broader exploration and experimentation to find truth.  There is a decent Q-learning example that allows the agent to start at many initial conditions, just click the topics-links near the bottom  (https://people.revoledu.com/kardi/tutorial/ReinforcementLearning/).
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T20:14:24.650Z
Comments.Text: Please, put your **specific question** in the title. "reinforcement learning with changing rewards" is not a question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards><reward-functions><reward-shaping>
Title: How to solve a reinforcement learning problem with changing rewards?
UserDisplayName: Zhenzhen Gong
ThreadText: # How to solve a reinforcement learning problem with changing rewards?
  
  (Asked by: username_0 on 2022-06-06)
  
  I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for the same $(s, a)$ pair can change with time too.
  
  In my problem, a UAV flies and visits several devices that would go through non-stationary environments. Each visit of the UAV can result in a performance improvement for the visited device. This happens gradually until a new environment appears for this device.
  
  So, the state is the time elapsed since the last time it visited each device. The action is the chosen device. The reward is the summation of the average cost of **the current visited device and estimated rewards of other unvisited devices**. This results in the fact that the UAV's reward can change. Most of the time, it increases gradually. Occasionally, it drops because of a new environment. Like a staircase going up, up, up, and suddenly down. Then it goes up and repeats. The time frequency of a new environment for each device is unique.
  
  I tried the above state and reward settings. When I use Actor-Critic to solve it and only use the reward of the current visited device, the UAV tends to visit one device only and ignores other devices. But when I add the reward of other unvisited devices, no convergence can be seen. In this case, the UAV tends to visit only one device occasionally, even when the reward is extremely low.
  
  My questions are:
  
  1. Am I setting the state and reward in the right way?
  2. What algorithm should I use here?
  
  **Tags:** reinforcement-learning, rewards, reward-functions, reward-shaping
  > **Comments:**
  > **username_1**: Sounds like it settles down too early.  It needs broader exploration and experimentation to find truth.  There is a decent Q-learning example that allows the agent to start at many initial conditions, just click the topics-links near the bottom  (https://people.revoledu.com/kardi/tutorial/ReinforcementLearning/).  
  > **username_2**: Please, put your **specific question** in the title. "reinforcement learning with changing rewards" is not a question.  
  
  ---
  
  ## 0 answers
  
  