Answers.Body: <p>I suppose it is both time and memory actually.</p>
  <p>In case you do not quantize the weights and leave them as is (and without gradient checkpointing etc.), when using LoRA you only have to do backprop for a subset of weights, which reduces the finetuning time.</p>
  <p>On the other had, you often find yourself with a consumer grade GPU and frequently forced to apply quantize the weights (sometimes even apply double quantization) and you have to dequantize them for the forward and backward passes, this will result in a performance drop.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-31T16:47:52.263Z
Answers.LastActivityDate: 2024-05-31T16:47:52.263Z
Answers.UserDisplayName: vanerk
Body: <p>I tried lora (on ViT) and i thought it would reduce the finetuning time, but it is same. is it right??</p>
  <p>I checked that the original network's weights are forzen correctly.</p>
  <p>Then LoRA's advantage is reduction of computing resource, not time?</p>
  
Tags: <fine-tuning><vision-transformer>
Title: Fintuning Time of LoRA
UserDisplayName: COTHE
ThreadText: # Fintuning Time of LoRA
  
  (Asked by: username_0 on 2024-05-31)
  
  I tried lora (on ViT) and i thought it would reduce the finetuning time, but it is same. is it right??
  
  I checked that the original network's weights are forzen correctly.
  
  Then LoRA's advantage is reduction of computing resource, not time?
  
  **Tags:** fine-tuning, vision-transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-31. Score: 0 | Accepted answer
  
  I suppose it is both time and memory actually.
  
  In case you do not quantize the weights and leave them as is (and without gradient checkpointing etc.), when using LoRA you only have to do backprop for a subset of weights, which reduces the finetuning time.
  
  On the other had, you often find yourself with a consumer grade GPU and frequently forced to apply quantize the weights (sometimes even apply double quantization) and you have to dequantize them for the forward and backward passes, this will result in a performance drop.
  
  
  ---
  
  