Body: <h2>Description</h2>
  <p>My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all features the way I though is correct. Most of them are &lt;0,1&gt;</p>
  <p>I have few neural networks, using different architectures.
  But they face same problems. They seems not to learn.</p>
  <h2>Model</h2>
  <p>In input I pass</p>
  <ul>
  <li><p>data input (2d time series data)</p>
  </li>
  <li><p>actor state (1 or 0)</p>
  <p>currently it is just 0 or 1, depending if actor has any asset</p>
  </li>
  </ul>
  <p>I predict 3 actions q-values.</p>
  <ul>
  <li>
  <ol>
  <li>Buy</li>
  </ol>
  </li>
  <li>
  <ol start="2">
  <li>Pass</li>
  </ol>
  </li>
  <li>
  <ol start="3">
  <li>Sell</li>
  </ol>
  </li>
  </ul>
  <p>Example model:</p>
  <p><a href="https://i.sstatic.net/wyXnc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wyXnc.png" alt="Example neural network picture" /></a></p>
  <h2>Parameters</h2>
  <p>I used <span class="math-container">$1e-5$</span> learning rate and batch of 2000.</p>
  <p>Discount: I used <span class="math-container">$0.98$</span> before, but picture below is <span class="math-container">$discount=0.1$</span></p>
  <h2>Using bellman equation for deep LR</h2>
  <p><span class="math-container">$Q(a,s) $</span> ← <span class="math-container">$reward + \gamma * max(Q(s'))$</span></p>
  <pre><code>    new_vector = rewards + discount * max_future_argq * (1 - dones.astype(int))
      for i, (ac, v) in enumerate(zip(actions, new_vector)):
          curr_qvals[i, ac] = v
  </code></pre>
  <h4>Training with new qvals</h4>
  <p><code>envs_states_arr</code> - env array of shape <code>(actors, timewindow, features)</code></p>
  <p><code>states</code> - actor states of shape <code>(actors, 1</code>)</p>
  <pre><code>history_ob = mod.fit([envs_states_arr, states], new_qvals, shuffle=True,
                               batch_size=mini_batchsize,
                               verbose=False)
  </code></pre>
  <h2>Reward function</h2>
  <p><code>discrete_stock</code> is hidden integer for given actor, it means amount of items actor has</p>
  <p><code>price</code> - normalized price in given state.</p>
  <p><code>BASE_PENALTY</code> <span class="math-container">$ = -1$</span></p>
  <p><code>INVALID_MOVE</code> <span class="math-container">$ = -10$</span></p>
  <p><code>False</code> and <code>True</code> are indicating valid moves just for me.</p>
  <pre><code>if action == 0:
      &quot;BUY&quot;
      if discrete_stock == 0:
          return -price, True
      else:
          return INVALID_MOVE, False
  
  elif action == 1:
      &quot;PASS&quot;
      return 3 * BASE_PENALTY, True
  
  elif action == 2:
      &quot;SELL&quot;
      if discrete_stock &lt;= 0:
          return INVALID_MOVE, False
  
      return price * 2, True
  </code></pre>
  <h2>More about problem</h2>
  <p>I think it is already small learning rate. Should I go even lower learning rate and bigger batch size?</p>
  <p>I am confused about this problem. You can see there are some <code>-10</code> rewards in plot, so some outputs had invalid move, yet I don't see model to set any actions in that direction.</p>
  <p><a href="https://i.sstatic.net/IFM7g.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IFM7g.png" alt="Training results" /></a></p>
  <h1>Is model learning?</h1>
  <p>Loss is going very low. So is my solution designed wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:33:22.673Z
Comments.Text: Thanks for editing. You show some code, but it is hard to interpret - and also it leaves the question of whether you have the wrong idea, or simply implemented it wrong. Instead,please explain the "game" the agent is playing, what the rules are etc.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:35:59.057Z
Comments.Text: I am however, concerned by your "hidden" variable which has large influence on expected reward. I can see that could be a major problem. Is it truly hidden from the agent? Does it change when agent buys/sells (and not change on pass) or is is set randomly on each time step?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:38:43.197Z
Comments.Text: Yes and no. Amount is hidden, but actor knows if it has any asset, so it sees state as 1 or 0. Also rules are simple. It can buy only 1 at the time, and gets reward relative to price.
Comments.UserDisplayName: Grzegorz Krug
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:42:12.400Z
Comments.Text: Initially I trained him like any other "Game" but now I removed end condition and do not use "done" samples, cause there is no really end to the game, it was end of  the session.
Comments.UserDisplayName: Grzegorz Krug
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:46:37.137Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147353/discussion-between-grzegorz-krug-and-neil-slater).
Comments.UserDisplayName: Grzegorz Krug
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T21:08:14.660Z
Comments.Text: @NeilSlater thanks again for help, I found the issue, I copied values from actions to rewards.... just code issue
Comments.UserDisplayName: Grzegorz Krug
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T07:24:50.670Z
Comments.Text: That makes sense, I couldn't identify a major mistake in your work in our chat. But I hope the talk around what you are doing was still useful
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T10:14:32.333Z
Comments.Text: It was! I should visit this stack more often
Comments.UserDisplayName: Grzegorz Krug
Tags: <deep-learning><deep-rl><dqn>
Title: Deep Q problem. Some q-Values are always greater
UserDisplayName: Grzegorz Krug
ThreadText: # Deep Q problem. Some q-Values are always greater
  
  (Asked by: username_0 on 2023-07-19)
  
  ## Description
  
  My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all features the way I though is correct. Most of them are <0,1>
  
  I have few neural networks, using different architectures.
  But they face same problems. They seems not to learn.
  
  ## Model
  
  In input I pass
  
  * data input (2d time series data)
  * actor state (1 or 0)
  
    currently it is just 0 or 1, depending if actor has any asset
  
  I predict 3 actions q-values.
  
  * 1. Buy
  * 2. Pass
  * 3. Sell
  
  Example model:
  
  [![Example neural network picture](https://i.sstatic.net/wyXnc.png)](https://i.sstatic.net/wyXnc.png)
  
  ## Parameters
  
  I used $1e-5$ learning rate and batch of 2000.
  
  Discount: I used $0.98$ before, but picture below is $discount=0.1$
  
  ## Using bellman equation for deep LR
  
  $Q(a,s) $ ← $reward + \gamma \* max(Q(s'))$
  
  ```
      new_vector = rewards + discount * max_future_argq * (1 - dones.astype(int))
      for i, (ac, v) in enumerate(zip(actions, new_vector)):
          curr_qvals[i, ac] = v
  
  ```
  
  #### Training with new qvals
  
  `envs_states_arr` - env array of shape `(actors, timewindow, features)`
  
  `states` - actor states of shape `(actors, 1`)
  
  ```
  history_ob = mod.fit([envs_states_arr, states], new_qvals, shuffle=True,
                               batch_size=mini_batchsize,
                               verbose=False)
  
  ```
  
  ## Reward function
  
  `discrete_stock` is hidden integer for given actor, it means amount of items actor has
  
  `price` - normalized price in given state.
  
  `BASE_PENALTY` $ = -1$
  
  `INVALID_MOVE` $ = -10$
  
  `False` and `True` are indicating valid moves just for me.
  
  ```
  if action == 0:
      "BUY"
      if discrete_stock == 0:
          return -price, True
      else:
          return INVALID_MOVE, False
  
  elif action == 1:
      "PASS"
      return 3 * BASE_PENALTY, True
  
  elif action == 2:
      "SELL"
      if discrete_stock <= 0:
          return INVALID_MOVE, False
  
      return price * 2, True
  
  ```
  
  ## More about problem
  
  I think it is already small learning rate. Should I go even lower learning rate and bigger batch size?
  
  I am confused about this problem. You can see there are some `-10` rewards in plot, so some outputs had invalid move, yet I don't see model to set any actions in that direction.
  
  [![Training results](https://i.sstatic.net/IFM7g.png)](https://i.sstatic.net/IFM7g.png)
  
  # Is model learning?
  
  Loss is going very low. So is my solution designed wrong?
  
  **Tags:** deep-learning, deep-rl, dqn
  
  > **Comments:**
  > **username_1**: Thanks for editing. You show some code, but it is hard to interpret - and also it leaves the question of whether you have the wrong idea, or simply implemented it wrong. Instead,please explain the "game" the agent is playing, what the rules are etc.  
  > **username_1**: I am however, concerned by your "hidden" variable which has large influence on expected reward. I can see that could be a major problem. Is it truly hidden from the agent? Does it change when agent buys/sells (and not change on pass) or is is set randomly on each time step?  
  > **username_0**: Yes and no. Amount is hidden, but actor knows if it has any asset, so it sees state as 1 or 0. Also rules are simple. It can buy only 1 at the time, and gets reward relative to price.  
  > **username_0**: Initially I trained him like any other "Game" but now I removed end condition and do not use "done" samples, cause there is no really end to the game, it was end of  the session.  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147353/discussion-between-username_0-username_0-and-username_1).  
  > **username_0**: @username_1 thanks again for help, I found the issue, I copied values from actions to rewards.... just code issue  
  > **username_1**: That makes sense, I couldn't identify a major mistake in your work in our chat. But I hope the talk around what you are doing was still useful  
  > **username_0**: It was! I should visit this stack more often  
  
  ---
  
  ## 0 answers
  
  