RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 8192, 1]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.
### System Info

tranformer version: 4.2.0
Huggingface hub:0.13.2
Python: Python 3.9.16
![image](https://user-images.githubusercontent.com/90728105/225866607-921911db-b0b2-4a4e-8278-6a8326c36241.png)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I do trainer.train() to fine-tune pertained long former for text summarization, I get the following error:

> /usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py:197: UserWarning: Error detected in BmmBackward0. Traceback of forward call that caused the error:
>   File "/usr/local/lib/python3.9/dist-packages/torch/autograd/function.py", line 267, in apply
>     return user_fn(self, *args)
>   File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 141, in backward
>     outputs = ctx.run_function(*detached_inputs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 1701, in custom_forward
>     return module(*inputs, is_global_attn, output_attentions)
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 873, in forward
>     attn_outputs = self.self_attn(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 695, in forward
>     self_outputs = self.longformer_self_attn(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 268, in forward
>     attn_output = self._compute_attn_output_with_global_indices(
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 578, in _compute_attn_output_with_global_indices
>     attn_output_only_global = torch.matmul(
>   File "/usr/local/lib/python3.9/dist-packages/torch/fx/traceback.py", line 57, in format_stack
>     return traceback.format_stack()
>  (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:114.)
>   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
> /usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py:197: UserWarning: 
> 
> Previous calculation was induced by CheckpointFunctionBackward. Traceback of forward call that induced the previous calculation:
>   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
>     exec(code, run_globals)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel_launcher.py", line 16, in <module>
>     app.launch_new_instance()
>   File "/usr/local/lib/python3.9/dist-packages/traitlets/config/application.py", line 992, in launch_instance
>     app.start()
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelapp.py", line 612, in start
>     self.io_loop.start()
>   File "/usr/local/lib/python3.9/dist-packages/tornado/platform/asyncio.py", line 215, in start
>     self.asyncio_loop.run_forever()
>   File "/usr/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
>     self._run_once()
>   File "/usr/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
>     handle._run()
>   File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
>     self._context.run(self._callback, *self._args)
>   File "/usr/local/lib/python3.9/dist-packages/tornado/ioloop.py", line 687, in <lambda>
>     lambda f: self._run_callback(functools.partial(callback, future))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/ioloop.py", line 740, in _run_callback
>     ret = callback()
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 821, in inner
>     self.ctx_run(self.run)
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 782, in run
>     yielded = self.gen.send(value)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 365, in process_one
>     yield gen.maybe_future(dispatch(*args))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
>     yield gen.maybe_future(handler(stream, idents, msg))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 543, in execute_request
>     self.do_execute(
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/ipkernel.py", line 306, in do_execute
>     res = shell.run_cell(code, store_history=store_history, silent=silent)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/zmqshell.py", line 536, in run_cell
>     return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 2854, in run_cell
>     result = self._run_cell(
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 2881, in _run_cell
>     return runner(coro)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
>     coro.send(None)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3057, in run_cell_async
>     has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3249, in run_ast_nodes
>     if (await self.run_code(code, result,  async_=asy)):
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3326, in run_code
>     exec(code_obj, self.user_global_ns, self.user_ns)
>   File "<ipython-input-120-3435b262f1ae>", line 1, in <module>
>     trainer.train()
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 888, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 1248, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 1277, in compute_loss
>     outputs = model(**inputs)
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 2190, in forward
>     outputs = self.led(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 2044, in forward
>     encoder_outputs = self.encoder(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 1705, in forward
>     layer_outputs = torch.utils.checkpoint.checkpoint(
>   File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
>     return CheckpointFunction.apply(function, preserve, *args)
>   File "/usr/local/lib/python3.9/dist-packages/torch/fx/traceback.py", line 57, in format_stack
>     return traceback.format_stack()
>  (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:121.)
>   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
> /usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py:197: UserWarning: Error detected in CheckpointFunctionBackward. Traceback of forward call that caused the error:
>   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
>     exec(code, run_globals)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel_launcher.py", line 16, in <module>
>     app.launch_new_instance()
>   File "/usr/local/lib/python3.9/dist-packages/traitlets/config/application.py", line 992, in launch_instance
>     app.start()
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelapp.py", line 612, in start
>     self.io_loop.start()
>   File "/usr/local/lib/python3.9/dist-packages/tornado/platform/asyncio.py", line 215, in start
>     self.asyncio_loop.run_forever()
>   File "/usr/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
>     self._run_once()
>   File "/usr/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
>     handle._run()
>   File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
>     self._context.run(self._callback, *self._args)
>   File "/usr/local/lib/python3.9/dist-packages/tornado/ioloop.py", line 687, in <lambda>
>     lambda f: self._run_callback(functools.partial(callback, future))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/ioloop.py", line 740, in _run_callback
>     ret = callback()
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 821, in inner
>     self.ctx_run(self.run)
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 782, in run
>     yielded = self.gen.send(value)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 365, in process_one
>     yield gen.maybe_future(dispatch(*args))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
>     yield gen.maybe_future(handler(stream, idents, msg))
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/kernelbase.py", line 543, in execute_request
>     self.do_execute(
>   File "/usr/local/lib/python3.9/dist-packages/tornado/gen.py", line 234, in wrapper
>     yielded = ctx_run(next, result)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/ipkernel.py", line 306, in do_execute
>     res = shell.run_cell(code, store_history=store_history, silent=silent)
>   File "/usr/local/lib/python3.9/dist-packages/ipykernel/zmqshell.py", line 536, in run_cell
>     return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 2854, in run_cell
>     result = self._run_cell(
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 2881, in _run_cell
>     return runner(coro)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
>     coro.send(None)
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3057, in run_cell_async
>     has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3249, in run_ast_nodes
>     if (await self.run_code(code, result,  async_=asy)):
>   File "/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py", line 3326, in run_code
>     exec(code_obj, self.user_global_ns, self.user_ns)
>   File "<ipython-input-120-3435b262f1ae>", line 1, in <module>
>     trainer.train()
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 888, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 1248, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 1277, in compute_loss
>     outputs = model(**inputs)
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 2190, in forward
>     outputs = self.led(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 2044, in forward
>     encoder_outputs = self.encoder(
>   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/usr/local/lib/python3.9/dist-packages/transformers/models/led/modeling_led.py", line 1705, in forward
>     layer_outputs = torch.utils.checkpoint.checkpoint(
>   File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
>     return CheckpointFunction.apply(function, preserve, *args)
>   File "/usr/local/lib/python3.9/dist-packages/torch/fx/traceback.py", line 57, in format_stack
>     return traceback.format_stack()
>  (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:114.)
>   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
> ---------------------------------------------------------------------------
> RuntimeError                              Traceback (most recent call last)
> <ipython-input-120-3435b262f1ae> in <module>
> ----> 1 trainer.train()
> 
> 6 frames
> /usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
>     195     # some Python versions print out the first line of a multi-line function
>     196     # calls in the traceback and some print out the last line
> --> 197     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
>     198         tensors, grad_tensors_, retain_graph, create_graph, inputs,
>     199         allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
> 
> RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 8192, 1]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

code : https://github.com/Tanya-11/experiment/blob/main/lonformer_experiment.ipynb


### Expected behavior

trainer.train() should run
Hi @Tanya-11, thanks for raising this issue! 

It's not possible to access the code in the link shared as it is private. Could you share a minimal code snippet to reproduce the error?  <|||||>> Hi @Tanya-11, thanks for raising this issue!
> 
> It's not possible to access the code in the link shared as it is private. Could you share a minimal code snippet to reproduce the error?

Hi @amyeroberts 
Pls find the link to my public[ github repo](https://github.com/Tanya-11/experiment/blob/main/lonformer_experiment.ipynb).
Thanks! <|||||>Hi @Tanya-11, thanks for sharing the link. 

I am able to run the example code if I set `gradient_checkpointing=False`. There have been recent updates to the LED model, including this one which resolves an [issue with gradient checkpointing](https://github.com/huggingface/transformers/pull/21840). Can you retry with the most recent release of transformers? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.