Answers.Body: <p>LoRA stands for &quot;low-rank adaptation&quot;, use a <span class="math-container">$d\times d$</span> matrix, and you get rid of the &quot;LoR&quot; part in the name (aka, that you have full rank)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-28T17:56:47.150Z
Answers.LastActivityDate: 2024-05-28T17:56:47.150Z
Answers.UserDisplayName: Alberto
Body: <p>I've learned that LoRA injects trainable rank decomposition 'matrices' into each layer.</p>
  <p>A matrix reduces dimension from d to r, and B matrix increases dimension from r to d.</p>
  <p>What I'm having trouble in understanding is that why it has this sturucture.</p>
  <p>I'm curious what happens if it has simple linear strucutre from dimension d to d.</p>
  <p>does the encoder - decoder structure gives some kind of feature learning??</p>
  
Tags: <neural-networks><fine-tuning>
Title: Why does LoRA has encoder-decoder structure?
UserDisplayName: COTHE
ThreadText: # Why does LoRA has encoder-decoder structure?
  
  (Asked by: username_0 on 2024-05-28)
  
  I've learned that LoRA injects trainable rank decomposition 'matrices' into each layer.
  
  A matrix reduces dimension from d to r, and B matrix increases dimension from r to d.
  
  What I'm having trouble in understanding is that why it has this sturucture.
  
  I'm curious what happens if it has simple linear strucutre from dimension d to d.
  
  does the encoder - decoder structure gives some kind of feature learning??
  
  **Tags:** neural-networks, fine-tuning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-28. Score: 0 | Accepted answer
  
  LoRA stands for "low-rank adaptation", use a $d\times d$ matrix, and you get rid of the "LoR" part in the name (aka, that you have full rank)
  
  
  ---
  
  