Answers.Body: <p>They all have different weight initializations. I think that the chance of gradient descent discovering the same local minima in such high dimensions is low.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T01:24:24.853Z
Answers.LastActivityDate: 2024-02-23T01:24:24.853Z
Answers.UserDisplayName: JobHunter69
Body: <p>How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ensure they don't &quot;converge&quot;?</p>
  <p>(My question is partly influenced by the notion of kernels in computer vision. Those kernels also focus on different aspects of the image, but they don't end up learning the same thing since they're specified ahead of time.)</p>
  
Tags: <deep-learning><transformer><attention>
Title: How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?
UserDisplayName: Tfovid
ThreadText: # How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?
  
  (Asked by: username_0 on 2024-01-27)
  
  How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ensure they don't "converge"?
  
  (My question is partly influenced by the notion of kernels in computer vision. Those kernels also focus on different aspects of the image, but they don't end up learning the same thing since they're specified ahead of time.)
  
  **Tags:** deep-learning, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-23. Score: 1
  
  They all have different weight initializations. I think that the chance of gradient descent discovering the same local minima in such high dimensions is low.
  
  
  ---
  
  