Answers.Body: <p>If I understand correctly, your question is whether it's possible to randomly initialize some layers in a neural network and not apply gradient descent to them.</p>
  <p>I believe the answer is yes. In synthesizer, they use a random matrix in the &quot;attention layer&quot; and did not train them, and still achieved reasonable results. Though I'm not sure how &quot;optimal&quot; that is. See here:
  <a href="https://arxiv.org/abs/2005.00743" rel="nofollow noreferrer">https://arxiv.org/abs/2005.00743</a></p>
  <p>I'm sure this applies to other networks (e.g. CNNs) as well. Is there a particular application you have in mind for this?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-27T21:17:14.447Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-27T20:04:14.363Z
Answers.LastActivityDate: 2022-08-27T20:04:14.363Z
Answers.UserDisplayName: PeaBrane
Body: <p>Is it possible to reach the optimal values ​​for the parameters by not applying the decreasing gradient in some layers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-27T00:51:14.710Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-22T00:16:23.677Z
Comments.Text: How will you set initial value of the kernel in the layer?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><deep-learning>
Title: Reach optimal values by not decreasing gradient
UserDisplayName: Faeze
ThreadText: # Reach optimal values by not decreasing gradient
  
  (Asked by: username_0 on 2022-08-27)
  
  Is it possible to reach the optimal values ​​for the parameters by not applying the decreasing gradient in some layers?
  
  **Tags:** neural-networks, deep-learning
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_2**: How will you set initial value of the kernel in the layer?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-08-27. Score: 1
  
  If I understand correctly, your question is whether it's possible to randomly initialize some layers in a neural network and not apply gradient descent to them.
  
  I believe the answer is yes. In synthesizer, they use a random matrix in the "attention layer" and did not train them, and still achieved reasonable results. Though I'm not sure how "optimal" that is. See here:
  <https://arxiv.org/abs/2005.00743>
  
  I'm sure this applies to other networks (e.g. CNNs) as well. Is there a particular application you have in mind for this?
  
  > **Comments:**
  > **username_1**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  