Answers.Body: <p>The output of randomly initialised neural networks will be random. I am not sure if I understand your question correctly but I would like to try and explain what OpenAI is trying to achieve with their RND Distillation approach.</p>
  
  <p>Any re-enforcement or supervised "learning" algorithm such as a neural network (NN) with back-propagation, genetic algorithm, partial swarm etc requires a fitness function. The fitness function "directs" the algorithm to a solution. </p>
  
  <p>The basic idea is that the algorithm is randomly initialised. The algorithm is run, and then parts of the algorithm are changed. In this case the weights of the NN.</p>
  
  <p>A number of algorithms exist to update the weights, the most common back propagation / gradient decent as used in the paper. To update the weights the output of the NN is compared to the optimum output. Sometimes the optimum is known. Other times it is not. In the case of a game the optimum output is not known. It is assumed that the more points gained the better the NN does. In this case the fitness function or policy is the sum of points gained when playing the game. But gradient descent needs an error (i.e difference between output and expected output), which does not exist when playing a game, because you don't know what the points should be. However the NN could play one round of the game score some points, then run the NN again on the next move to determine a prediction of future points. The difference between the points gained and the prediction is then the error, which gradient descent can use. [Hope this makes sense]</p>
  
  <p>This is fine for games like pong, a NN could play the game pong randomly and score point at each prediction, which will direct gradient descent into updating the weights to maximise points.</p>
  
  <p>However, games such as Montezuma’s Revenge, requires a lot of complex actions before points are gained. If the points stay 0, how does gradient descent determine if the weights should be decreased or increased?</p>
  
  <p>Instead, the authors initialise a second random neural network which remains the same for the duration of training. As mentioned above the output of a randomly initialised neural network is random. </p>
  
  <p>The output of the actual NN, and world state is then passed through the random NN (RND), which will return a random value. The NN is then required to not only determine a game action but also determine the result of the RND. </p>
  
  <p>The policy function is extended from just points to also predict the result of the RND. The greater the prediction error the more bonus points are allocated to the policy function. This encourages gradient descent to change the weights of the NN to explore unseen parts of the game as this increases the prediction error of the RND which increases points. At the same time gradient descent must update the weights of the NN to predict the RND.</p>
  
  <p>I hope this answers your question.</p>
  
  <p>In summary randomly initialised NN include multilayered perceptrons or convolutional networks will behave randomly with respect to their inputs. The RND network is randomly set once at the star of training, and used to basically remind the NN undergoing training that it has done this before do not do this again - i.e. rewards the NN for doing something new (exploration).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-07T18:40:05.477Z
Answers.Text: Here is someone that explained it better than me :) https://towardsdatascience.com/curiosity-in-deep-reinforcement-learning-understanding-random-network-distillation-747b322e2403
Answers.UserDisplayName: Jason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-07T23:42:43.147Z
Answers.Text: My question is more about how chaotic they are: like do small changes in inputs in the input yield small changes in outputs? (I updated the question to clarify this)
  
  I would like to see some experiments that show this, or some theoretical analysis.
  
  What about them makes them good for curiosity?  Like why not just use a single layer of random weights that you pass the input through?  My guess is that they have some interesting properties.
Answers.UserDisplayName: matwilso
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-07T23:43:23.227Z
Answers.Text: Oh I found this Twitter thread that I was thinking about: https://twitter.com/LiamFedus/status/1020324447348318208 when asking this question
Answers.UserDisplayName: matwilso
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-08T06:29:17.213Z
Answers.Text: Ah :) ok that is an interesting question. I don't know of any experiments that have done this. But I suspect the effect input has on output would be dependent on the architecture. In a small NN with few inputs, a small change in one input may make a huge difference on the output. In a large network I think the weights would make  a bigger difference. I don't really know the answer to this. It is an interesting question!
Answers.UserDisplayName: Jason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-07T18:38:43.093Z
Answers.LastActivityDate: 2018-12-07T18:38:43.093Z
Answers.UserDisplayName: Jason
Answers.Body: <p>In the <a href="https://arxiv.org/pdf/1808.04355.pdf" rel="nofollow noreferrer">Large-Scale Study of Curiosity-Driven Learning</a> paper (the prequel to the Random Network Distillation work), in their discussion of Random Features, they reference 3 papers that discuss this:</p>
  
  <ol>
  <li>K. Jarrett, K. Kavukcuoglu, Y. LeCun, et al. <a href="http://yann.lecun.com/exdb/publis/pdf/jarrett-iccv-09.pdf" rel="nofollow noreferrer">What is the Best Multi-Stage Architecture for Object Recognition?</a></li>
  <li>A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng. <a href="http://www.robotics.stanford.edu/~ang/papers/nipsdlufl10-RandomWeights.pdf" rel="nofollow noreferrer">On Random Weights and Unsupervised Feature Learning</a></li>
  <li>Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang. <a href="https://arxiv.org/pdf/1412.7149.pdf" rel="nofollow noreferrer">Deep Fried Convnets</a></li>
  </ol>
  
  <p>I just briefly glanced over these.  For now, one interesting idea from [2] is to use randomly initialized networks for architecture search.  To evaluate the architecture for the task, you don't have to train it; you can just randomly initialize it and measure its performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-01T18:42:39.130Z
Answers.LastActivityDate: 2019-01-01T18:42:39.130Z
Answers.UserDisplayName: matwilso
Body: <p>I am wondering how the output of randomly initialized MLPs and ConvNets behave with respect to their inputs.  Can anyone point to some analysis or explanation of this?</p>
  
  <p>I am curious about this because in the <a href="https://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/" rel="nofollow noreferrer">Random Network Distillation work from OpenAI</a>, they use the output of randomly initialized network to generate intrinsic reward for exploration.  It seems that this assumes that similar states will produce similar outputs of the random network.  Is this generally the case?  </p>
  
  <p>Do small changes in input yield small changes in output, or is it more <a href="https://en.wikipedia.org/wiki/Chaos_theory" rel="nofollow noreferrer">chaotic</a>?  Do they have other interesting properties?</p>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks>
Title: How do randomly initialized neural networks behave?
UserDisplayName: matwilso
ThreadText: # How do randomly initialized neural networks behave?
  
  (Asked by: username_0 on 2018-12-06)
  
  I am wondering how the output of randomly initialized MLPs and ConvNets behave with respect to their inputs. Can anyone point to some analysis or explanation of this?
  
  I am curious about this because in the [Random Network Distillation work from OpenAI](https://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/), they use the output of randomly initialized network to generate intrinsic reward for exploration. It seems that this assumes that similar states will produce similar outputs of the random network. Is this generally the case?
  
  Do small changes in input yield small changes in output, or is it more [chaotic](https://en.wikipedia.org/wiki/Chaos_theory)? Do they have other interesting properties?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-12-07. Score: 4
  
  The output of randomly initialised neural networks will be random. I am not sure if I understand your question correctly but I would like to try and explain what OpenAI is trying to achieve with their RND Distillation approach.
  
  Any re-enforcement or supervised "learning" algorithm such as a neural network (NN) with back-propagation, genetic algorithm, partial swarm etc requires a fitness function. The fitness function "directs" the algorithm to a solution.
  
  The basic idea is that the algorithm is randomly initialised. The algorithm is run, and then parts of the algorithm are changed. In this case the weights of the NN.
  
  A number of algorithms exist to update the weights, the most common back propagation / gradient decent as used in the paper. To update the weights the output of the NN is compared to the optimum output. Sometimes the optimum is known. Other times it is not. In the case of a game the optimum output is not known. It is assumed that the more points gained the better the NN does. In this case the fitness function or policy is the sum of points gained when playing the game. But gradient descent needs an error (i.e difference between output and expected output), which does not exist when playing a game, because you don't know what the points should be. However the NN could play one round of the game score some points, then run the NN again on the next move to determine a prediction of future points. The difference between the points gained and the prediction is then the error, which gradient descent can use. [Hope this makes sense]
  
  This is fine for games like pong, a NN could play the game pong randomly and score point at each prediction, which will direct gradient descent into updating the weights to maximise points.
  
  However, games such as Montezuma’s Revenge, requires a lot of complex actions before points are gained. If the points stay 0, how does gradient descent determine if the weights should be decreased or increased?
  
  Instead, the authors initialise a second random neural network which remains the same for the duration of training. As mentioned above the output of a randomly initialised neural network is random.
  
  The output of the actual NN, and world state is then passed through the random NN (RND), which will return a random value. The NN is then required to not only determine a game action but also determine the result of the RND.
  
  The policy function is extended from just points to also predict the result of the RND. The greater the prediction error the more bonus points are allocated to the policy function. This encourages gradient descent to change the weights of the NN to explore unseen parts of the game as this increases the prediction error of the RND which increases points. At the same time gradient descent must update the weights of the NN to predict the RND.
  
  I hope this answers your question.
  
  In summary randomly initialised NN include multilayered perceptrons or convolutional networks will behave randomly with respect to their inputs. The RND network is randomly set once at the star of training, and used to basically remind the NN undergoing training that it has done this before do not do this again - i.e. rewards the NN for doing something new (exploration).
  
  > **Comments:**
  > **username_1**: Here is someone that explained it better than me :) https://towardsdatascience.com/curiosity-in-deep-reinforcement-learning-understanding-random-network-distillation-747b322e2403  
  > **username_0**: My question is more about how chaotic they are: like do small changes in inputs in the input yield small changes in outputs? (I updated the question to clarify this)  I would like to see some experiments that show this, or some theoretical analysis.  What about them makes them good for curiosity?  Like why not just use a single layer of random weights that you pass the input through?  My guess is that they have some interesting properties.  
  > **username_0**: Oh I found this Twitter thread that I was thinking about: https://twitter.com/LiamFedus/status/1020324447348318208 when asking this question  
  > **username_1**: Ah :) ok that is an interesting question. I don't know of any experiments that have done this. But I suspect the effect input has on output would be dependent on the architecture. In a small NN with few inputs, a small change in one input may make a huge difference on the output. In a large network I think the weights would make  a bigger difference. I don't really know the answer to this. It is an interesting question!  
  
  ---
  
  ### Answer by username_0 on 2019-01-01. Score: 0
  
  In the [Large-Scale Study of Curiosity-Driven Learning](https://arxiv.org/pdf/1808.04355.pdf) paper (the prequel to the Random Network Distillation work), in their discussion of Random Features, they reference 3 papers that discuss this:
  
  1. K. Jarrett, K. Kavukcuoglu, Y. LeCun, et al. [What is the Best Multi-Stage Architecture for Object Recognition?](http://yann.lecun.com/exdb/publis/pdf/jarrett-iccv-09.pdf)
  2. A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng. [On Random Weights and Unsupervised Feature Learning](http://www.robotics.stanford.edu/~ang/papers/nipsdlufl10-RandomWeights.pdf)
  3. Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang. [Deep Fried Convnets](https://arxiv.org/pdf/1412.7149.pdf)
  
  I just briefly glanced over these. For now, one interesting idea from [2] is to use randomly initialized networks for architecture search. To evaluate the architecture for the task, you don't have to train it; you can just randomly initialize it and measure its performance.
  
  
  ---
  
  