<h3 dir="auto">System Info</h3>
<p dir="auto">The PEFT methods freeze the bulk of the transformer, apart from an external module.</p>
<p dir="auto">When I enable gradient checkpointing and train with these models or even if I simply freeze an embedding layer of a normal model, training breaks. So this problem is not specific to PEFT: gradient_checkpointing + frozen first parameter = Error</p>
<p dir="auto">But if I do</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" for n, param in model.named_parameters():
            param.requires_grad = True
            break"><pre class="notranslate"><code class="notranslate"> for n, param in model.named_parameters():
            param.requires_grad = True
            break
</code></pre></div>
<p dir="auto">it trains successfully. So it seems like there is a check if the first parameter has a gradient. Ideally, I would not have to set the first parameter (embedding) to True, as I want the whole model including embeddings frozen.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  warnings.warn(&quot;None of the inputs have requires_grad=True. Gradients will be None&quot;)
/admin/miniconda3/envs/peft/lib/python3.10/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(&quot;None of the inputs have requires_grad=True. Gradients will be None&quot;)

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /admin/peft/model/model_training/trainer.py:48 │
│ 0 in &lt;module&gt;                                                                                    │
│                                                                                                  │
│                                                                      │
│                                                                                                  │
│ /admin/peft/model/model_training/trainer.py:47 │
│ 4 in main                                                                                        │
│                                                                                                  │
│   471 │   │   compute_metrics=partial(compute_metrics, metrics=metrics, preprocess_fns=preproc   │
│   472 │   │   preprocess_logits_for_metrics=preprocess_logits_for_metrics,                       │
│   473 │   )                                                                                      │
│ ❱ 474 │   trainer.train(resume_from_checkpoint=training_conf.resume_from_checkpoint)             │
│   475 │   trainer.save_model()                                                                   │
│   476 │   tokenizer.save_pretrained(output_dir)                                                  │
│   477                                                                                            │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:1639 in train                                                                         │
│                                                                                                  │
│   1636 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1637 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1638 │   │   )                                                                                 │
│ ❱ 1639 │   │   return inner_training_loop(                                                       │
│   1640 │   │   │   args=args,                                                                    │
│   1641 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1642 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:1906 in _inner_training_loop                                                          │
│                                                                                                  │
│   1903 │   │   │   │   │   with model.no_sync():                                                 │
│   1904 │   │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                  │
│   1905 │   │   │   │   else:                                                                     │
│ ❱ 1906 │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                      │
│   1907 │   │   │   │                                                                             │
│   1908 │   │   │   │   if (                                                                      │
│   1909 │   │   │   │   │   args.logging_nan_inf_filter                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:2668 in training_step                                                                 │
│                                                                                                  │
│   2665 │   │   │   │   scaled_loss.backward()                                                    │
│   2666 │   │   elif self.deepspeed:                                                              │
│   2667 │   │   │   # loss gets scaled under gradient_accumulation_steps in deepspeed             │
│ ❱ 2668 │   │   │   loss = self.deepspeed.backward(loss)                                          │
│   2669 │   │   else:                                                                             │
│   2670 │   │   │   loss.backward()                                                               │
│   2671                                                                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/uti │
│ ls/nvtx.py:11 in wrapped_fn                                                                      │
│                                                                                                  │
│    8 │   function call.&quot;&quot;&quot;                                                                       │
│    9 │   def wrapped_fn(*args, **kwargs):                                                        │
│   10 │   │   get_accelerator().range_push(func.__qualname__)                                     │
│ ❱ 11 │   │   ret_val = func(*args, **kwargs)                                                     │
│   12 │   │   get_accelerator().range_pop()                                                       │
│   13 │   │   return ret_val                                                                      │
│   14                                                                                             │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/engine.py:1974 in backward                                                                  │
│                                                                                                  │
│   1971 │   │   if self.zero_optimization():                                                      │
│   1972 │   │   │   self.optimizer.is_gradient_accumulation_boundary = self.is_gradient_accumula  │
│   1973 │   │   │   )                                                                             │
│ ❱ 1974 │   │   │   self.optimizer.backward(loss, retain_graph=retain_graph)                      │
│   1975 │   │   elif self.amp_enabled():                                                          │
│   1976 │   │   │   # AMP requires delaying unscale when inside gradient accumulation boundaries  │
│   1977 │   │   │   # https://nvidia.github.io/apex/advanced.html#gradient-accumulation-across-i  │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/zero/stage_1_and_2.py:2028 in backward                                                      │
│                                                                                                  │
│   2025 │   │   │   scaled_loss = self.external_loss_scale * loss                                 │
│   2026 │   │   │   scaled_loss.backward()                                                        │
│   2027 │   │   else:                                                                             │
│ ❱ 2028 │   │   │   self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)            │
│   2029 │                                                                                         │
│   2030 │   def check_overflow(self, partition_gradients=True):                                   │
│   2031 │   │   self._check_overflow(partition_gradients)                                         │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/fp16/loss_scaler.py:54 in backward                                                          │
│                                                                                                  │
│    51 │                                                                                          │
│    52 │   def backward(self, loss, retain_graph=False):                                          │
│    53 │   │   scaled_loss = loss * self.loss_scale                                               │
│ ❱  54 │   │   scaled_loss.backward(retain_graph=retain_graph)                                    │
│    55                                                                                            │
│    56                                                                                            │
│    57 class LossScaler(LossScalerBase):                                                          │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/torch/_tensor │
│ .py:488 in backward                                                                              │
│                                                                                                  │
│    485 │   │   │   │   create_graph=create_graph,                                                │
│    486 │   │   │   │   inputs=inputs,                                                            │
│    487 │   │   │   )                                                                             │
│ ❱  488 │   │   torch.autograd.backward(                                                          │
│    489 │   │   │   self, gradient, retain_graph, create_graph, inputs=inputs                     │
│    490 │   │   )                                                                                 │
│    491                                                                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/torch/autogra │
│ d/__init__.py:197 in backward                                                                    │
│                                                                                                  │
│   194 │   # The reason we repeat same the comment below is that                                  │
│   195 │   # some Python versions print out the first line of a multi-line function               │
│   196 │   # calls in the traceback and some print out the last line                              │
│ ❱ 197 │   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the bac   │
│   198 │   │   tensors, grad_tensors_, retain_graph, create_graph, inputs,                        │
│   199 │   │   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to ru   │
│   200                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn"><pre class="notranslate"><code class="notranslate">  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
/admin/miniconda3/envs/peft/lib/python3.10/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /admin/peft/model/model_training/trainer.py:48 │
│ 0 in &lt;module&gt;                                                                                    │
│                                                                                                  │
│                                                                      │
│                                                                                                  │
│ /admin/peft/model/model_training/trainer.py:47 │
│ 4 in main                                                                                        │
│                                                                                                  │
│   471 │   │   compute_metrics=partial(compute_metrics, metrics=metrics, preprocess_fns=preproc   │
│   472 │   │   preprocess_logits_for_metrics=preprocess_logits_for_metrics,                       │
│   473 │   )                                                                                      │
│ ❱ 474 │   trainer.train(resume_from_checkpoint=training_conf.resume_from_checkpoint)             │
│   475 │   trainer.save_model()                                                                   │
│   476 │   tokenizer.save_pretrained(output_dir)                                                  │
│   477                                                                                            │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:1639 in train                                                                         │
│                                                                                                  │
│   1636 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1637 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1638 │   │   )                                                                                 │
│ ❱ 1639 │   │   return inner_training_loop(                                                       │
│   1640 │   │   │   args=args,                                                                    │
│   1641 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1642 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:1906 in _inner_training_loop                                                          │
│                                                                                                  │
│   1903 │   │   │   │   │   with model.no_sync():                                                 │
│   1904 │   │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                  │
│   1905 │   │   │   │   else:                                                                     │
│ ❱ 1906 │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                      │
│   1907 │   │   │   │                                                                             │
│   1908 │   │   │   │   if (                                                                      │
│   1909 │   │   │   │   │   args.logging_nan_inf_filter                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/transformers/ │
│ trainer.py:2668 in training_step                                                                 │
│                                                                                                  │
│   2665 │   │   │   │   scaled_loss.backward()                                                    │
│   2666 │   │   elif self.deepspeed:                                                              │
│   2667 │   │   │   # loss gets scaled under gradient_accumulation_steps in deepspeed             │
│ ❱ 2668 │   │   │   loss = self.deepspeed.backward(loss)                                          │
│   2669 │   │   else:                                                                             │
│   2670 │   │   │   loss.backward()                                                               │
│   2671                                                                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/uti │
│ ls/nvtx.py:11 in wrapped_fn                                                                      │
│                                                                                                  │
│    8 │   function call."""                                                                       │
│    9 │   def wrapped_fn(*args, **kwargs):                                                        │
│   10 │   │   get_accelerator().range_push(func.__qualname__)                                     │
│ ❱ 11 │   │   ret_val = func(*args, **kwargs)                                                     │
│   12 │   │   get_accelerator().range_pop()                                                       │
│   13 │   │   return ret_val                                                                      │
│   14                                                                                             │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/engine.py:1974 in backward                                                                  │
│                                                                                                  │
│   1971 │   │   if self.zero_optimization():                                                      │
│   1972 │   │   │   self.optimizer.is_gradient_accumulation_boundary = self.is_gradient_accumula  │
│   1973 │   │   │   )                                                                             │
│ ❱ 1974 │   │   │   self.optimizer.backward(loss, retain_graph=retain_graph)                      │
│   1975 │   │   elif self.amp_enabled():                                                          │
│   1976 │   │   │   # AMP requires delaying unscale when inside gradient accumulation boundaries  │
│   1977 │   │   │   # https://nvidia.github.io/apex/advanced.html#gradient-accumulation-across-i  │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/zero/stage_1_and_2.py:2028 in backward                                                      │
│                                                                                                  │
│   2025 │   │   │   scaled_loss = self.external_loss_scale * loss                                 │
│   2026 │   │   │   scaled_loss.backward()                                                        │
│   2027 │   │   else:                                                                             │
│ ❱ 2028 │   │   │   self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)            │
│   2029 │                                                                                         │
│   2030 │   def check_overflow(self, partition_gradients=True):                                   │
│   2031 │   │   self._check_overflow(partition_gradients)                                         │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/deepspeed/run │
│ time/fp16/loss_scaler.py:54 in backward                                                          │
│                                                                                                  │
│    51 │                                                                                          │
│    52 │   def backward(self, loss, retain_graph=False):                                          │
│    53 │   │   scaled_loss = loss * self.loss_scale                                               │
│ ❱  54 │   │   scaled_loss.backward(retain_graph=retain_graph)                                    │
│    55                                                                                            │
│    56                                                                                            │
│    57 class LossScaler(LossScalerBase):                                                          │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/torch/_tensor │
│ .py:488 in backward                                                                              │
│                                                                                                  │
│    485 │   │   │   │   create_graph=create_graph,                                                │
│    486 │   │   │   │   inputs=inputs,                                                            │
│    487 │   │   │   )                                                                             │
│ ❱  488 │   │   torch.autograd.backward(                                                          │
│    489 │   │   │   self, gradient, retain_graph, create_graph, inputs=inputs                     │
│    490 │   │   )                                                                                 │
│    491                                                                                           │
│                                                                                                  │
│ /admin//miniconda3/envs/peft/lib/python3.10/site-packages/torch/autogra │
│ d/__init__.py:197 in backward                                                                    │
│                                                                                                  │
│   194 │   # The reason we repeat same the comment below is that                                  │
│   195 │   # some Python versions print out the first line of a multi-line function               │
│   196 │   # calls in the traceback and some print out the last line                              │
│ ❱ 197 │   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the bac   │
│   198 │   │   tensors, grad_tensors_, retain_graph, create_graph, inputs,                        │
│   199 │   │   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to ru   │
│   200                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre></div>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">from transformers import AutoModel</p>
<p dir="auto">from peft import LoraConfig,  get_peft_model<br>
model = AutoModel.from_pretrained('decapoda-research/llama-7b-hf')<br>
config = LoraConfig(<br>
r=16,<br>
lora_alpha=32,<br>
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],<br>
lora_dropout=0.05,<br>
bias="none",<br>
task_type="CAUSAL_LM",<br>
)<br>
model = get_peft_model(model, config)</p>
<p dir="auto">and then proceed to train with gradient_checkpointing enabled.</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Gradient checkpointing shouldn't affect whether a subset of parameters are frozen. As PEFT models are increasingly popular as well as gradient_checkpointing  it makes sense to get to the bottom of this bug.</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto">I think it'd be nice to have a simple way to preprocess the logits before caching them for computing metrics.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">When the <code class="notranslate">Trainer</code> <code class="notranslate">compute_metrics</code> are set, during evaluation the logits are accumulated (some in GPU memory, for <code class="notranslate">args.eval_accumulation_steps</code> steps; all in RAM). For some models, it will almost certainly lead to out of memory problems.</p>
<p dir="auto">For instance, for a language model, this means storing in RAM a tensor of size [eval ds size, sequence length, vocab size].</p>
<p dir="auto">In many cases, what is needed to compute metrics is just some reduction of the logits. For example: <code class="notranslate">logits.argmax(dim=-1)</code>.</p>
<p dir="auto">I know I can subclass <code class="notranslate">Trainer</code> for this and redefine <code class="notranslate">evaluation_loop</code>, just wanted to know if you'd consider a more generic solution that prevents everyone that needs the feature from duplicating the rest of the code of <code class="notranslate">evaluation_loop</code>. I've seen more people running into the same issue. For instance:</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="741166156" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/8476" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/8476/hovercard" href="https://github.com/huggingface/transformers/issues/8476">#8476</a><br>
<a href="https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941" rel="nofollow">https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941</a><br>
<a href="https://discuss.huggingface.co/t/cuda-out-of-memory-during-evaluation-but-training-is-fine/1783/4" rel="nofollow">https://discuss.huggingface.co/t/cuda-out-of-memory-during-evaluation-but-training-is-fine/1783/4</a></p>
<h2 dir="auto">Your contribution</h2>
<p dir="auto">I was thinking about something like adding a <code class="notranslate">preprocess_logits_for_metrics</code> parameter to <code class="notranslate">TrainingArguments</code> of type Callable</p>
<p dir="auto">If you don't set the parameter, the default is None and everything would work as always. If you set it, the logits are passed to <code class="notranslate">args.preprocess_logits_for_metrics</code> and its output is what's cached.</p>
<p dir="auto">The main modification would be this in <code class="notranslate">Trainer.evaluation_loop</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Update containers on host
...
 if logits is not None:
    logits = self._pad_across_processes(logits)
    logits = self._nested_gather(logits)
    if self.args.preprocess_logits_for_metrics is not None:
        logits = self.args.preprocess_logits_for_metrics(logits)
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)"><pre class="notranslate"><code class="notranslate"># Update containers on host
...
 if logits is not None:
    logits = self._pad_across_processes(logits)
    logits = self._nested_gather(logits)
    if self.args.preprocess_logits_for_metrics is not None:
        logits = self.args.preprocess_logits_for_metrics(logits)
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
</code></pre></div>
<p dir="auto">Do you think it's worth it? If you do, I can submit a PR.</p>
<p dir="auto">I tag <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> because I think he's worked quite a lot with the training loop, but I'm open to receive feedback from anyone.</p>