Answers.Body: <p>Normally, an adapter to perform binary classification might be similar to what you've described, but with the classifier and without the relu before the sigmoid (sigmoid after relu will output &gt;=0.5).  You may also want to try using the output embedding layer's input rather than its output (vocab logits) if possible, and remove the sigmoid and use BCEWithLogitsLoss (it can be more numerically stable).</p>
  <p>For this sort of model, I think it's unlikely that the gradients will vanish like that if the adapter and training procedure is correct.  As the adapter is currently, for base model logit outputs less than 0, which in some models could consistently be the case, the gradient will be 0 and backpropagate as 0s no matter the loss.  If this is the issue that you're facing, then you would find 0.5s for the forward probabilities.  Additionally, <code>dropout_output[:, -1, :]</code> looks like it's sampling the last output token logits.  You probably want to check that the last output token is not degenerate in some way (like almost always being <code>.</code> or here a pad token).  If neither of these are the case perhaps there is some issue with your loss function, labels or inputs, however, I think even if those were incorrect, they are not likely to be causing 0 gradients here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-20T18:15:53.057Z
Answers.Text: I see. I am trying out `BCEWithLogitsLoss`, would you recommend removing the `ReLU` along with `Sigmoid` from the custom class?
Answers.UserDisplayName: kms
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-21T05:59:59.387Z
Answers.Text: Yes, with BCEWithLogitsLoss I would recommend removing both the relu and the sigmoid.  Dropout should also be removed if it's the last layer before the loss function.  It may be reasonable to also just include the single linear layer for now, but if you want multiple layers, you can run something like (linear, relu, optionally dropout) x n, linear -> BCEWithLogitsLoss, where n might be something like 1-3 for a simple example.  The output dimension of the last linear layer will be the number of binary outputs you want.  During inference you could check whether the output is greater than 0.
Answers.UserDisplayName: Greg S
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-20T07:54:32.843Z
Answers.LastActivityDate: 2024-06-20T07:54:32.843Z
Answers.UserDisplayName: Greg S
Body: <p>I am fine-tuning a <code>mistral-7b</code> with Hugging Face <code>peft</code> and quantization. In my training loop, I am printing the gradient values for each batch, which seem a bit unusual.</p>
  <pre><code># Print gradients
  for name, param in model_init.named_parameters():
        if param.grad is not None:
             print(f'Gradient for {name}: {param.grad.norm()}')
  </code></pre>
  <p>I am trying to understanding, why all the gradients values are <code>0s</code> except for the #1 iteration (starting from 0th).</p>
  <pre><code>iteration 0
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0
  
  iteration 1
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight: 0.0142822265625
  Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight: 3.953125
  Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight: 0.185546875
  
  iteration n
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0
  </code></pre>
  <p>Isn't this a bit unusual? Is it the vanishing gradient problem?</p>
  <p>For context,</p>
  <pre><code># custom class
  class BinaryClassification(nn.Module):
      
      def __init__(self, base_model):
          super().__init__()
          self.base_model = base_model
          self.dropout = nn.Dropout(0.05)
          #self.classifier = nn.Linear(hidden_size, 1)
          self.relu = nn.ReLU()
          self.sigmoid = nn.Sigmoid()
  
      def forward(self, x):
          outputs = self.base_model(x)
          dropout_output = self.dropout(outputs.logits)
          relu_output = self.relu(dropout_output[:, -1, :])
          probs = self.sigmoid(relu_output) # Apply sigmoid to logits to get probabilities
          #print('forward probs', probs)
          return probs
  
  model_init = BinaryClassification(peft_model)
  
  # optimizer
  criterion = torch.nn.BCELoss()
  optimizer = torch.optim.AdamW(model_init.parameters(), lr=0.001, eps=1e-08, weight_decay=0.001)
  scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=True)
  
  # Loss
  def calc_loss_batch(input_batch, target_batch):
      
      output = model_init(input_batch)
      
      # Reshape target to match the shape of probs
      target_batch = target_batch.unsqueeze(1)
      
      if output.shape != target_batch.shape:
          raise Exception(&quot;Shape mismatch between input logits and target label&quot;)
      
      # Logits of last output token
      loss = criterion(output, target_batch)
      
      return loss
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <deep-learning><pytorch><fine-tuning><vanishing-gradient-problem>
Title: Why are all the gradients values 0 except for the first iteration?
UserDisplayName: kms
ThreadText: # Why are all the gradients values 0 except for the first iteration?
  
  (Asked by: username_0 on 2024-06-20)
  
  I am fine-tuning a mistral-7b with Hugging Face peft and quantization. In my training loop, I am printing the gradient values for each batch, which seem a bit unusual.
  # Print gradients
  for name, param in model_init.named_parameters():
        if param.grad is not None:
             print(f'Gradient for {name}: {param.grad.norm()}')
  
  I am trying to understanding, why all the gradients values are 0s except for the #1 iteration (starting from 0th).
  iteration 0
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0
  
  iteration 1
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight: 0.0142822265625
  Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight: 3.953125
  Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight: 0.185546875
  
  iteration n
  ...
  ...
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
  Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0
  
  Isn't this a bit unusual? Is it the vanishing gradient problem?
  For context,
  # custom class
  class BinaryClassification(nn.Module):
      
      def __init__(self, base_model):
          super().__init__()
          self.base_model = base_model
          self.dropout = nn.Dropout(0.05)
          #self.classifier = nn.Linear(hidden_size, 1)
          self.relu = nn.ReLU()
          self.sigmoid = nn.Sigmoid()
  
      def forward(self, x):
          outputs = self.base_model(x)
          dropout_output = self.dropout(outputs.logits)
          relu_output = self.relu(dropout_output[:, -1, :])
          probs = self.sigmoid(relu_output) # Apply sigmoid to logits to get probabilities
          #print('forward probs', probs)
          return probs
  
  model_init = BinaryClassification(peft_model)
  
  # optimizer
  criterion = torch.nn.BCELoss()
  optimizer = torch.optim.AdamW(model_init.parameters(), lr=0.001, eps=1e-08, weight_decay=0.001)
  scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=True)
  
  # Loss
  def calc_loss_batch(input_batch, target_batch):
      
      output = model_init(input_batch)
      
      # Reshape target to match the shape of probs
      target_batch = target_batch.unsqueeze(1)
      
      if output.shape != target_batch.shape:
          raise Exception("Shape mismatch between input logits and target label")
      
      # Logits of last output token
      loss = criterion(output, target_batch)
      
      return loss
  ```
  
  **Tags:** deep-learning, pytorch, fine-tuning, vanishing-gradient-problem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-20. Score: 1
  
  Normally, an adapter to perform binary classification might be similar to what you've described, but with the classifier and without the relu before the sigmoid (sigmoid after relu will output >=0.5). You may also want to try using the output embedding layer's input rather than its output (vocab logits) if possible, and remove the sigmoid and use BCEWithLogitsLoss (it can be more numerically stable).
  
  For this sort of model, I think it's unlikely that the gradients will vanish like that if the adapter and training procedure is correct. As the adapter is currently, for base model logit outputs less than 0, which in some models could consistently be the case, the gradient will be 0 and backpropagate as 0s no matter the loss. If this is the issue that you're facing, then you would find 0.5s for the forward probabilities. Additionally, `dropout_output[:, -1, :]` looks like it's sampling the last output token logits. You probably want to check that the last output token is not degenerate in some way (like almost always being `.` or here a pad token). If neither of these are the case perhaps there is some issue with your loss function, labels or inputs, however, I think even if those were incorrect, they are not likely to be causing 0 gradients here.
  
  > **Comments:**
  > **username_0**: I see. I am trying out `BCEWithLogitsLoss`, would you recommend removing the `ReLU` along with `Sigmoid` from the custom class?  
  > **username_1**: Yes, with BCEWithLogitsLoss I would recommend removing both the relu and the sigmoid.  Dropout should also be removed if it's the last layer before the loss function.  It may be reasonable to also just include the single linear layer for now, but if you want multiple layers, you can run something like (linear, relu, optionally dropout) x n, linear -> BCEWithLogitsLoss, where n might be something like 1-3 for a simple example.  The output dimension of the last linear layer will be the number of binary outputs you want.  During inference you could check whether the output is greater than 0.  
  
  ---
  
  