SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7f06bfae6b30> returned NULL without setting an error> ```
> ```
> import pickle
> from transformers import AutoModelForCausalLM
> 
> pickle.dumps(AutoModelForCausalLM)
> ```
> 
> I think it's comes from the fact those are autogenerated.

Thanks for your help, but I tested based on your modification in #12654, a new problem arises：

@stas00 @patrickvonplaten, @LysandreJik


Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/media/cfs/gonglixing/9Nctl/gpt_v2/run_clm_v3.py", line 509, in init_process
    fn(rank, size)
  File "/media/cfs/gonglixing/9Nctl/gpt_v2/run_clm_v3.py", line 456, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/media/cfs/gonglixing/9Nctl/opensource/transformers-master/src/transformers/trainer.py", line 1275, in train
    tr_loss += self.training_step(model, inputs)
  File "/media/cfs/gonglixing/9Nctl/opensource/transformers-master/src/transformers/trainer.py", line 1778, in training_step
    self.scaler.scale(loss).backward()
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7f06bfae6b30> returned NULL without setting an error

_Originally posted by @lancekung in https://github.com/huggingface/transformers/issues/12621#issuecomment-878738997_
Hmm, somehow this issue has never been addressed. 

In such cases you will have a better luck reporting torch-land issues to https://github.com/pytorch/pytorch/issues as chances are low we will have the required understanding.

I tried to google the exception and only found this to be relevant:
https://discuss.pytorch.org/t/autograd-vague-error-returned-null-without-setting-an-error/112781/6
Are you by chance too using apex's amp?

Someone reported that building their own version of pytorch solved the problem. So perhaps you could try to switch to an older or newer pytorch and see if the problem goes away?

On my setup (pt-1.9.0)
```
python -c "import pickle; from transformers import AutoModelForCausalLM; pickle.dumps(AutoModelForCausalLM)"
```
it works w/o a problem (that is if that's the code that caused the error in OP).<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.