Answers.Body: <h2>Yes and No.</h2>
  <p>Yes, it <strong>is</strong> possible to achieve that result. But instead of a self-optimizing Neural Network, I'd recommend another approach:</p>
  <h2>1. Don't let training time interfere.</h2>
  <p>If you are trying to train the agent DURING the environment runtime, that's probably the problem.
  Training time is usually much bigger than evaluation time. And deployed models usually don't train, so it won't be a problem in production.</p>
  <p>You can do 2 things about that:</p>
  <h3>1.1 &quot;Pause&quot; the game during training.</h3>
  <p>It might look like &quot;cheating&quot;, but in your agent's point of view, they are not actually playing during train time. And once again, it's just simulating how it would behave in production.</p>
  <p>But if you can't pause it:</p>
  <h3>1.2 Disable training during runtime.</h3>
  <p>Store all states and decisions. Wait until the game is over and <strong>then</strong> you train the whole batch.</p>
  <blockquote>
  <p>Professional chess players don't try to learn during a blitz challenge. But they do study their own games later to learn from their mistakes.</p>
  </blockquote>
  <h2>2. Optimizing hyper-parameters for speed.</h2>
  <p>You could tweak some hyper-parameters (like the size of your NN) looking for a faster model. Keep in mind it would still be an algorithm that always runs in a fixed time, but you might find some way to make it always</p>
  <h2>2.1 Using Machine Learning for this Optimization</h2>
  <p>There are some meta-learning techniques and other methods like <a href="https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies" rel="nofollow noreferrer">NEAT</a>, that can automate your search for a simple effective topology. NEAT already rewards the simplest architectures, penalizing complexity (which is usually attached to speed), but you could also force it to consider running time specifically.</p>
  <h2>3. Another Network for Another Task</h2>
  <p>You could a make another short network for deciding if the next move should be accurate or fast. Based on this result, it will choose between precision or speed. This choice could be by flagging a parameter (like branch prediction) or even running a whole new algorithm:</p>
  <pre><code>NeedForSpeed = TimeEstimation(state)
  #Sorry, I couldn't resist the pun!
  
  if NeedForSpeed &gt; 0.8
    decision = agent.instantReactionDecisionTree(state)
  elif NeedForSpeed &gt; 0.5
    decision = agent.decideStandard(state, branchPrediction=True )
  elif NeedForSpeed &gt; 0.2
    decision = agent.decideStandard(state)
  else:
    decision = agent.DeepNN(state)
  </code></pre>
  <h2>Bonus: Use Other ML algorithms</h2>
  <p>Some algorithms have explicit parameters that directly affect the tradeoff for <em>time x precision</em>.</p>
  <p>For instance, MCST (Monte Carlos Search Tree) can keep running forever until it explores all possibilities, but it usually finishes before, giving you the best solution found <em>so far</em>.</p>
  <p>So, one possibility would be trying some other method instead of Neural Networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-22T20:41:00.170Z
Answers.LastActivityDate: 2021-08-22T20:41:00.170Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I am experimenting with OpenAI Gym and reinforcement learning. As far as I understood, the environment is waiting for the agent to make a decision, so it's a sequential operation like this:</p>
  
  <pre><code>decision = agent.decide(state)
  state, reward, done = environment.act(decision)
  agent.train(state, reward)
  </code></pre>
  
  <p>Doing it in this sequential way, the Markov property is fulfilled: the new state is a result of the old state and the action. However, a lot of games will not wait for the player to make a decision. The game will continue to run and perhaps, the action comes too late.</p>
  
  <p>Has it been observed or is it even possible that a neuronal network adjusts its weights so that the PC computes the result faster and thus makes the "better" decision? E.g. one AI beats another because it is faster.</p>
  
  <p>Before posting an answer like "there are always the same amount of calculations, so it's impossible", please be aware that there is caching (1st level cache versus RAM), <a href="https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array">branch prediction</a> and maybe other stuff.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-23T00:33:03.280Z
Comments.Text: It's important to note that neural networks are usually trained with gradient descent or other learning algorithms (which are usually iterative/numerical algorithms that use data to find some target function, which will hopefully be represented by the neural network). So, I am not sure if it's a good idea to talk about "self-optimization" (in the title).
Comments.UserDisplayName: nbro
Tags: <neural-networks><reinforcement-learning><open-ai><markov-property>
Title: Is a neural network able to optimize itself for speed?
UserDisplayName: Thomas Weller
ThreadText: # Is a neural network able to optimize itself for speed?
  
  (Asked by: username_0 on 2020-01-31)
  
  I am experimenting with OpenAI Gym and reinforcement learning. As far as I understood, the environment is waiting for the agent to make a decision, so it's a sequential operation like this:
  
  ```
  decision = agent.decide(state)
  state, reward, done = environment.act(decision)
  agent.train(state, reward)
  
  ```
  
  Doing it in this sequential way, the Markov property is fulfilled: the new state is a result of the old state and the action. However, a lot of games will not wait for the player to make a decision. The game will continue to run and perhaps, the action comes too late.
  
  Has it been observed or is it even possible that a neuronal network adjusts its weights so that the PC computes the result faster and thus makes the "better" decision? E.g. one AI beats another because it is faster.
  
  Before posting an answer like "there are always the same amount of calculations, so it's impossible", please be aware that there is caching (1st level cache versus RAM), [branch prediction](https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array) and maybe other stuff.
  
  **Tags:** neural-networks, reinforcement-learning, open-ai, markov-property
  
  > **Comments:**
  > **username_1**: It's important to note that neural networks are usually trained with gradient descent or other learning algorithms (which are usually iterative/numerical algorithms that use data to find some target function, which will hopefully be represented by the neural network). So, I am not sure if it's a good idea to talk about "self-optimization" (in the title).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-22. Score: 3
  
  ## Yes and No.
  
  Yes, it **is** possible to achieve that result. But instead of a self-optimizing Neural Network, I'd recommend another approach:
  
  ## 1. Don't let training time interfere.
  
  If you are trying to train the agent DURING the environment runtime, that's probably the problem.
  Training time is usually much bigger than evaluation time. And deployed models usually don't train, so it won't be a problem in production.
  
  You can do 2 things about that:
  
  ### 1.1 "Pause" the game during training.
  
  It might look like "cheating", but in your agent's point of view, they are not actually playing during train time. And once again, it's just simulating how it would behave in production.
  
  But if you can't pause it:
  
  ### 1.2 Disable training during runtime.
  
  Store all states and decisions. Wait until the game is over and **then** you train the whole batch.
  
  > Professional chess players don't try to learn during a blitz challenge. But they do study their own games later to learn from their mistakes.
  
  ## 2. Optimizing hyper-parameters for speed.
  
  You could tweak some hyper-parameters (like the size of your NN) looking for a faster model. Keep in mind it would still be an algorithm that always runs in a fixed time, but you might find some way to make it always
  
  ## 2.1 Using Machine Learning for this Optimization
  
  There are some meta-learning techniques and other methods like [NEAT](https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies), that can automate your search for a simple effective topology. NEAT already rewards the simplest architectures, penalizing complexity (which is usually attached to speed), but you could also force it to consider running time specifically.
  
  ## 3. Another Network for Another Task
  
  You could a make another short network for deciding if the next move should be accurate or fast. Based on this result, it will choose between precision or speed. This choice could be by flagging a parameter (like branch prediction) or even running a whole new algorithm:
  
  ```
  NeedForSpeed = TimeEstimation(state)
  #Sorry, I couldn't resist the pun!
  
  if NeedForSpeed > 0.8
    decision = agent.instantReactionDecisionTree(state)
  elif NeedForSpeed > 0.5
    decision = agent.decideStandard(state, branchPrediction=True )
  elif NeedForSpeed > 0.2
    decision = agent.decideStandard(state)
  else:
    decision = agent.DeepNN(state)
  
  ```
  
  ## Bonus: Use Other ML algorithms
  
  Some algorithms have explicit parameters that directly affect the tradeoff for *time x precision*.
  
  For instance, MCST (Monte Carlos Search Tree) can keep running forever until it explores all possibilities, but it usually finishes before, giving you the best solution found *so far*.
  
  So, one possibility would be trying some other method instead of Neural Networks.
  
  
  ---
  
  