Body: <p>I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.</p>
  <p>The problem is designing a device with maximum efficiency. So we simulated the problem as follows. There is a 4x4 grid (we defined a 4x4 matrix) and the elements of this matrix can be either 0 or 1 (value 0 means &quot;air&quot; and 1 means a certain material in reality), so there is a 2^16 possible configurations for this matrix. Our agent starts from the top left corner of this matrix and has 5 possible actions: move up, down, left, right and flip (which means flipping a 0 to 1 or vice versa). Based on flipping action, we get a new configuration and each configuration has an efficiency (which is calculated by maxwell equations in the background).</p>
  <p>Our goal is to find the best configuration so that the efficiency of the device is maximum.</p>
  <p>So far we have tried many reward functions and non of them seemed to work at all! I will mention some of them:</p>
  <ol>
  <li><p><code>reward = current_efficiency - previous_efficiency (the efficiency is being calculated in each time step)</code></p>
  </li>
  <li>
  <pre><code> if current_efficiency &gt; previous_efficiency:
      reward = current_efficiency
      previous_efficiency = current_efficiency
  
  </code></pre>
  </li>
  <li>
  <pre><code> diff = current_efficiency - previous_efficiency
   if diff &gt; 0:
      reward = 1
   else:
      reward = -2
  
  </code></pre>
  </li>
  </ol>
  <p>and some other variations. Nothing is working for our problem and the agent doesn't learn at all! So far, we have used different approaches to DQN and also A2C method and so far no positive feedback. We tried different definitions of states as well, but we don't think that is the problem.</p>
  <p>So, can somebody maybe help me with this? It would be a huge help!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-17T21:25:06.480Z
Comments.Text: how expensive is it to calculate the efficiency? 2^16 doesn't seem like a _huge_ number, could you not just brute force it? other than that, what are you defining as a state, which position in the grid the agent is in?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T02:03:41.420Z
Comments.Text: To find the best configuration seems more like a search rather than a reinforcement learning one ?
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-22T09:20:53.250Z
Comments.Text: @DavidIreland The problem is that this is a test case. My actual Problem has 10 ^120 possible states. My state consist of the configurations of the grid ( a matirx of 0s and 1s) and another matrix which has all the values = 0 except where currently the agent is. There the value =1. Then these two are flatend and added together.
Comments.UserDisplayName: Hossein
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-22T09:23:05.797Z
Comments.Text: @calveeen yes, but we want to find the best policy with reinforcement learning. Thats the whole idea about the project.
Comments.UserDisplayName: Hossein
Tags: <machine-learning><reinforcement-learning><deep-learning><dqn><reward-design>
Title: Designing a reward function for my reinforcement learning problem
UserDisplayName: Hossein
ThreadText: # Designing a reward function for my reinforcement learning problem
  
  (Asked by: username_0 on 2020-06-17)
  
  I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.
  The problem is designing a device with maximum efficiency. So we simulated the problem as follows. There is a 4x4 grid (we defined a 4x4 matrix) and the elements of this matrix can be either 0 or 1 (value 0 means "air" and 1 means a certain material in reality), so there is a 2^16 possible configurations for this matrix. Our agent starts from the top left corner of this matrix and has 5 possible actions: move up, down, left, right and flip (which means flipping a 0 to 1 or vice versa). Based on flipping action, we get a new configuration and each configuration has an efficiency (which is calculated by maxwell equations in the background).
  Our goal is to find the best configuration so that the efficiency of the device is maximum.
  So far we have tried many reward functions and non of them seemed to work at all! I will mention some of them:
  
  reward = current_efficiency - previous_efficiency (the efficiency is being calculated in each time step)
  
  
   if current_efficiency > previous_efficiency:
      reward = current_efficiency
      previous_efficiency = current_efficiency
  
  
  
  
   diff = current_efficiency - previous_efficiency
   if diff > 0:
      reward = 1
   else:
      reward = -2
  
  
  
  
  and some other variations. Nothing is working for our problem and the agent doesn't learn at all! So far, we have used different approaches to DQN and also A2C method and so far no positive feedback. We tried different definitions of states as well, but we don't think that is the problem.
  So, can somebody maybe help me with this? It would be a huge help!
  
  **Tags:** machine-learning, reinforcement-learning, deep-learning, dqn, reward-design
  > **Comments:**
  > **username_1**: how expensive is it to calculate the efficiency? 2^16 doesn't seem like a _huge_ number, could you not just brute force it? other than that, what are you defining as a state, which position in the grid the agent is in?  
  > **username_2**: To find the best configuration seems more like a search rather than a reinforcement learning one ?  
  > **username_0**: @DavidIreland The problem is that this is a test case. My actual Problem has 10 ^120 possible states. My state consist of the configurations of the grid ( a matirx of 0s and 1s) and another matrix which has all the values = 0 except where currently the agent is. There the value =1. Then these two are flatend and added together.  
  > **username_0**: @calveeen yes, but we want to find the best policy with reinforcement learning. Thats the whole idea about the project.  
  
  ---
  
  ## 0 answers
  
  