Answers.Body: <p>This turns out to be less about the loss function and more about the approach. There's a number of them implemented in the <a href="https://github.com/tensorflow/agents" rel="nofollow noreferrer"><code>tf_agents</code></a> package.</p>
  <p>Choosing the DDPG agent for outputting the increments and implementing a <code>PyEnvironment</code> to run the simulation and return the highest reached output as a reward seems to work just fine. Very similar to the blackjack example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-12T11:17:14.977Z
Answers.LastActivityDate: 2021-12-12T11:17:14.977Z
Answers.UserDisplayName: viraptor
Body: <p>I'm trying to train a function for a industrial-process-control-like system. This is my first attempt at a custom training, so feel free to point out any invalid assumptions.</p>
  <p>I've got one input and one controlled output, which I'm trying to optimise. I've reduced the problem with some values normalisation to:</p>
  <ul>
  <li><p>the first half of input looks like a <code>sin()</code> raise from 0 to max value then dropoff - with lots of noise on top (let's say up to +/-10% at each measurement)</p>
  </li>
  <li><p>I don't know what the max is, but it's roughly predictable (input goes from 0 to between 0.5 and 2)</p>
  </li>
  <li><p>the output cannot go down (well, it can by a tiny bit, but I'd ignore that here), and cannot go higher than the input value</p>
  </li>
  <li><p>the goal is to get the output value as high as possible</p>
  </li>
  </ul>
  <p>Currently the best non-NN approach I've got is to start a few % below input and at each step run <code>output = A*input + (1-A)*previous_output</code>, so the result looks like this (input in blue, output in orange)</p>
  <p><a href="https://i.sstatic.net/OvGtm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OvGtm.png" alt="tracking_input" /></a></p>
  <p>I wanted to check if some RNN can improve on this, so I'm planning to check an LSTM doing this instead. I'm struggling to come up with a loss calculation which is viable for training here.</p>
  <p>I considered making the input and output for the network an absolute change from the previous value (or input as <code>input-last_output</code> difference), then as the loss using some kind of inverted value of <code>sum(output changes)</code> from step 0 to the crossover point period to reward higher maximums while ignoring the distance. (so discarding anything that happens past the crossover)</p>
  <p>But... that doesn't relate to the input really, so <code>tensorflow</code> wouldn't be able to train based on this, if I understand it correctly. Am I going in the wrong direction? Are there some known ways to solve this problem?</p>
  
Tags: <recurrent-neural-networks><loss>
Title: Writing a loss function for "how far can this output be pushed"
UserDisplayName: viraptor
ThreadText: # Writing a loss function for "how far can this output be pushed"
  
  (Asked by: username_0 on 2021-12-09)
  
  I'm trying to train a function for a industrial-process-control-like system. This is my first attempt at a custom training, so feel free to point out any invalid assumptions.
  
  I've got one input and one controlled output, which I'm trying to optimise. I've reduced the problem with some values normalisation to:
  
  * the first half of input looks like a `sin()` raise from 0 to max value then dropoff - with lots of noise on top (let's say up to +/-10% at each measurement)
  * I don't know what the max is, but it's roughly predictable (input goes from 0 to between 0.5 and 2)
  * the output cannot go down (well, it can by a tiny bit, but I'd ignore that here), and cannot go higher than the input value
  * the goal is to get the output value as high as possible
  
  Currently the best non-NN approach I've got is to start a few % below input and at each step run `output = A*input + (1-A)*previous_output`, so the result looks like this (input in blue, output in orange)
  
  [![tracking_input](https://i.sstatic.net/OvGtm.png)](https://i.sstatic.net/OvGtm.png)
  
  I wanted to check if some RNN can improve on this, so I'm planning to check an LSTM doing this instead. I'm struggling to come up with a loss calculation which is viable for training here.
  
  I considered making the input and output for the network an absolute change from the previous value (or input as `input-last_output` difference), then as the loss using some kind of inverted value of `sum(output changes)` from step 0 to the crossover point period to reward higher maximums while ignoring the distance. (so discarding anything that happens past the crossover)
  
  But... that doesn't relate to the input really, so `tensorflow` wouldn't be able to train based on this, if I understand it correctly. Am I going in the wrong direction? Are there some known ways to solve this problem?
  
  **Tags:** recurrent-neural-networks, loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-12-12. Score: 0 | Accepted answer
  
  This turns out to be less about the loss function and more about the approach. There's a number of them implemented in the [`tf_agents`](https://github.com/tensorflow/agents) package.
  
  Choosing the DDPG agent for outputting the increments and implementing a `PyEnvironment` to run the simulation and return the highest reached output as a reward seems to work just fine. Very similar to the blackjack example.
  
  
  ---
  
  