--fp16 fine-tuning appears to be taking more memory (4.3.0). 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.4.0-62-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: 4x A100-SXM4-40GB
- Using distributed or parallel set-up in script?: Yes


### Who can help
@alexorona @stas00 @sgugger 

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] my own modified scripts: (give details below)

Official trainer, optionally modified by adding "model.parallelize() " after loading.  (Results shown with and without). 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Regular seq2seq on data. 

Run script:
```
export BS=1; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1,2,3 ./finetune_trainer.py --model_name_or_path t5-large --output_dir output_dir --adam_eps 1e-06 --data_dir xsum --fp16 \
--do_train --learning_rate 3e-5 \
--logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 \
--overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS \
--warmup_steps 5 \

```

## Brief summary

1. When fine tuning T5, I'm observing memory usage to increase when using --fp16, though not as much as previously reported in #8403 . 
2. (Optional) Possibly related: I'm trying to squeeze T5-11B in 4x40GB A100s using model parallelism.  I seemed to be able to do it yesterday on 4.1.1 with a sequence length of 128, and I remember observing a fairly moderate seqlength vs memory usage dependence (as expected from the comment here ( https://github.com/huggingface/transformers/issues/8771#issuecomment-764058315 ), though I'm not an expert and I'm not sure if this increase only applies >512 tokens, and if what I saw yesterday was a fluke/error on my part somewhere).  Today on a fresh env/pull I'm not observing this dependence (though I'm not sure why -- it might be my issue -- data is reported at the bottom side).  

## To reproduce

Steps to reproduce the behavior:

1. 4.3.0, runscript as above, run with and without --fp16 option.  Different model sizes (and with/without model.parallelize() added, since I wasn't sure if that was the issue)

## Data
Below are three cases of memory usage with/without --fp16: 
1. with model.parallelize()
2. without model.parallelize()   (but GPUs still visible -- extra info, I thought it was interesting it still takes up extra memory on the other GPUs)
3. without model.parallelize()   (only 1 GPU visible)
- 
```
*** WITH MODEL.PARALLELIZE() ***

t5-3b, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+                                                                                                                                                           
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |                                                                                                                                                           
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    77W / 400W |  13598MiB / 40537MiB |     33%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   29C    P0    80W / 400W |  12874MiB / 40537MiB |     25%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   24C    P0    81W / 400W |  12874MiB / 40537MiB |      4%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   25C    P0    80W / 400W |  12874MiB / 40537MiB |     23%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

WITH --fp16: (takes more memory)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   27C    P0   108W / 400W |  15138MiB / 40537MiB |      6%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   28C    P0    99W / 400W |  14214MiB / 40537MiB |      9%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   23C    P0    85W / 400W |  14214MiB / 40537MiB |     12%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   25C    P0    92W / 400W |  14216MiB / 40537MiB |     11%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+


*** WITHOUT MODEL.PARALLELIZE, but all GPUs still visible ( CUDA_VISIBLE_DEVICES=0,1,2,3 ) ***
t5-large, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    93W / 400W |  20362MiB / 40537MiB |      1%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   27C    P0    78W / 400W |   6046MiB / 40537MiB |      3%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   22C    P0    78W / 400W |   6046MiB / 40537MiB |      3%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   24C    P0    79W / 400W |   6022MiB / 40537MiB |      7%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

t5-large, --max_source_length 128 --max_target_length 128
WITH --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    91W / 400W |  20318MiB / 40537MiB |      2%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   27C    P0    80W / 400W |   7304MiB / 40537MiB |      4%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   23C    P0    78W / 400W |   7304MiB / 40537MiB |      5%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   24C    P0    79W / 400W |   7280MiB / 40537MiB |      5%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+



*** WITHOUT MODEL.PARALLELIZE, ONLY 1 GPU VISIBLE ( CUDA_VISIBLE_DEVICES=0 ) ***

t5-large, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   29C    P0   101W / 400W |  13790MiB / 40537MiB |     32%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   26C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   21C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   23C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

t5-large, --max_source_length 128 --max_target_length 128
WITH --fp16 (more memory)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0   101W / 400W |  15012MiB / 40537MiB |     42%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   26C    P0    70W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   21C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   23C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

```

WRT sequence length vs data dependence, in a model parallel setting, I am observing today:
(varying --max_source_length and --max_target_length )

model, seq length, gpu0, gpu1, gpu2, gpu3

t5-large, 32 seq length, 5.7GB / 4.7GB / 4.7GB / 4.7GB
t5-large, 64 seq length, 5.7GB / 4.7GB / 4.7GB / 4.7GB
t5-large, 128 seq length, 5.8GB / 4.8GB / 4.8GB / 4.8GB
t5-large, 512 seq length, 6.0GB / 5.2GB / 5.2GB / 5.2GB

t5-3b, 64 seq length, 15.2GB / 14.3GB / 14.3GB / 14.3GB
t5-3b, 128 seq length, 15.2GB / 14.3GB / 14.3GB / 14.3GB
t5-3b, 256 seq length, 15.5GB / 14.7GB / 14.7GB / 14.7GB
t5-3b, 512 seq length, 16.2GB / 15.2GB / 15.2GB / 15.2GB

Essentially very minimal change in RAM requirements vs sequence length.  Though perhaps I have misconfigured something here. 


## Expected behavior

1. Less memory usage with --fp16  (should it be about half? suggested from https://github.com/huggingface/transformers/issues/8403#issuecomment-725562117 )
2. (Optional) Nominally, smaller sequence length models taking up significantly less memory?

Note that the memory as reported by nvidia-smi is not a perfectly reliable source (@stas00 could explain better than me why), the reliable metric is the batch size at which you get OOM. It is known on our side that FP16 does not save any memory for this particular script (I was investigating a general memory regression for this script that I fixed today and could see the memory being the same with or without FP16) but only for that script (memory is indeed saved on `run_glue` or other maintained examples). It seems to have been there for at least three months, so it's not a recent regression, I think it was always like this.

Didn't have time to investigate the source yet. Maybe it's something in the seq2seq models (this is on T5 here, on my side I noticed the usage being the same for mBART) or something in the script itself. Nevertheless, it's an issue to tackle indeed. I wonder if it only appears in a distributed setting or one GPU already, should run some tests to check that tomorrow.<|||||>> Note that the memory as reported by nvidia-smi is not a perfectly reliable source

## How to reliably use nvidia-smi/pynvml to measure memory used by your application.

1. when you just started the program and do `torch.ones(1)` you will see your `nvidia-smi` reporting a usage of 0.5-1.5GB depending on the card - this is the memory that CUDA allocates for its kernels. So this is not the memory used by your application. Therefore in any memory benchmark I first do `torch.ones(1)` to force the kernel preloading. And then you can somewhat rely on `nvidia-smi`
2. but it won't show you any cached by pytorch memory, so these reports can be quite meaningless. Therefore you have to call `gc.collect(); torch.cuda.empty_cache()` before you take a snapshot using `nvidia-smi`. `gc.collect()` forces garbage collection - it's not immediate in python when you deleted some variable or existed a function.
3. it's totally unreliable if you have multiple processes using the same gpu

In general for memory benchmarking it's better to use pynvml, since it's easier to use it. But it has the exact same issues as  `nvidia-smi`.

If you were to follow the above 3 rules exactly, then you can use `nvidia-smi`/`pynvml` to reliably measure memory. 

Otherwise, use torch.cuda memory functions which give you exact numbers in any situation. https://pytorch.org/docs/stable/cuda.html#memory-management<|||||>I have been seeing this fp16 behavior for many months, but we blamed it on my hardware. Since I have one old 1070 card and one new but not fully supported 3090. Waiting for cuda-11.2 support.

Do you notice any difference if you use apex instead of the native amp?

DeepSpeed implements their own fp16.

> Less memory usage with --fp16 (should it be about half? suggested from https://github.com/huggingface/transformers/issues/8403#issuecomment-725562117

Well, that was a huge bug in pytorch, related to autocast/fp16 but it has been fixed in pt-nightly and pt-1.7.1 - though I won't be surprised if there are still some bugs there - this is new in pytorch. That's why I suggest to test with apex.

Also you might want to try to disable `with autocast()`, perhaps you are hitting another caching bug - it will be slower since now fp16 will have to be reconverted many times, but you will be able to see if perhaps the memory overhead is due to caching still. 

It's in 3 places, grep for `autocast`:
```
src/transformers/trainer.py:            with autocast():
src/transformers/trainer.py:                with autocast():
src/transformers/trainer_seq2seq.py:                with autocast():
```<|||||>Thanks both -- 

> I have been seeing this fp16 behavior for many months, but we blamed it on my hardware. Since I have one old 1070 card and one new but not fully supported 3090. Waiting for cuda-11.2 support.
> 
> Do you notice any difference if you use apex instead of the native amp?

The original numbers (above) are with apex -- single-GPU --fp16 gives 15.0GB.  It looks like if I uninstall nvidia-apex it reduces to 13.8GB. 

> > Less memory usage with --fp16 (should it be about half? suggested from [#8403 (comment)](https://github.com/huggingface/transformers/issues/8403#issuecomment-725562117)
> 
> Well, that was a huge bug in pytorch, related to autocast/fp16 but it has been fixed in pt-nightly and pt-1.7.1 - though I won't be surprised if there are still some bugs there - this is new in pytorch. That's why I suggest to test with apex.
> 
> Also you might want to try to disable `with autocast()`, perhaps you are hitting another caching bug - it will be slower since now fp16 will have to be reconverted many times, but you will be able to see if perhaps the memory overhead is due to caching still.
> 
> It's in 3 places, grep for `autocast`:
> 
> ```
> src/transformers/trainer.py:            with autocast():
> src/transformers/trainer.py:                with autocast():
> src/transformers/trainer_seq2seq.py:                with autocast():
> ```

It looks like disabling 'with autocast()' in those 3 places also brings the single-GPU --fp16 memory on T5-Large from 15.0gb down to 13.8gb (the same as without the --fp16 option) -- so they're at parity in terms of memory in that case. 

> DeepSpeed implements their own fp16.

Just for completeness I tried it with DeepSpeed, and in the single-GPU settling I'm seeing 10.8GB with CPU-offloading enabled (about 3gb savings), and ~21.5GB without offloading (significantly higher... I'm not sure I know enough about DeepSpeed to know whether that would be expected, or whether it may be a config error on my part). (EDIT: And DeepSpeed, CPU offload, with 4 GPUs, appears to use 12.1-12.4GB per GPU). 


On my last (optional) bit in the original post -- just as a sanity check, any sense of whether the memory allocation vs sequence length looks as expected, or whether there should be much larger differences as sequence length increases/decreases?  (I'm trying to get a sense of whether it's likely for me to fit T5-11b in these 4x40gb cards in the near term, or whether I'll need to wait for the full model parallelism/fp16/deep speed offloading, but these new models are new territory for me) <|||||>Sleeping on it I would like to amend my first statement. The components on GPU memory are the following:
- the model weights
- the forward activations saved for gradient computation
- the gradients
- the optimizer state

If we look at what's happening with FP16 training (mixed precision) we have:
- the model in full precision so no memory saved there
- the forward activations saved for gradient computation are in mixed precision
- the gradients are computed in mixed precision *but* converted to full precision for the update, so no saving there
- the optimizer state is in full precision as all the updates are done in full precision

So the saving only happen for the forward activations saved for the backward computation, and there is a slight overhead because the gradients are properly stored both in half and full precision. (This is probably over-simplified but I think it's enough to explain what follows.)

Now let's look at a simple text-classification fine-tuning on 2 GPUs (I'm giving the command for reference):
```
export BS=16
python -m torch.distributed.launch \
    --nproc_per_node 2 examples/text-classification/run_glue.py \
    --model_name_or_path bert-base-cased \
    --task_name mrpc \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size $BS \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/mrpc \
    --overwrite_output_dir \
    --fp16
```
Since the only savings we get are in the model activations saved for the backward passed, it's logical that the bigger those activations are, the bigger the saving will be. If we try different batch sizes, I indeed get (this is with nvidia-smi so not completely reliable as said above but it will be a fair comparison):
| batch size | without --fp16 | with --fp16 | FP16 savings |
|:-:|:-:|:-:|:-:|
| 8 | 4247 | 4163 | 84 |
| 16 | 4971 | 4793 | 178 |
| 32 | 6827 | 6207 | 620 |
| 64 | 10037 | 8061 | 1976 |

So there is only a real memory saving if we train at a high batch size (and it's not half) and at batch sizes lower than 8, you actually get a bigger memory footprint (because of the overhead mentioned above). The gain for FP16 training is that in each of those cases, the training with the flag `--fp16` is twice as fast, which does require every tensor to have every dimension be a multiple of 8 (so if your batch size is not a multiple of 8, you won't get that speed-up, and the script `finetune_trainer.py` does not pad the tensors to a sequence length that is a multiple of 8).

TL;DR: FP16 with apex or AMP will only give you some memory savings with a reasonably high batch size.<|||||>Ahhh, very helpful, thanks.  So right now --fp16 is mostly for speed, and since most things are stored in full precision, there are essentially no expected memory savings at BS=1. 

Is storing everything at fp16 in the long term plans, or are there technical reasons why that's not a good idea? (e.g. significant degradation in task performance?) <|||||>You can't do the full training in FP16, it would not converge, which is why there is this mixed precision approach. Maybe DeepSpeed integrates further optimizations and helps to save more memory.<|||||>@sgugger, this is such a clear and well done foray into fp16 performance, let's not get it lost in the sea of Issues. 

I was thinking we should start a new doc  `performance.md` (markdown pretty please) where we discuss each of these issues. And you have just written a perfect entry on fp16.<|||||>> The original numbers (above) are with apex -- single-GPU --fp16 gives 15.0GB. It looks like if I uninstall nvidia-apex it reduces to 13.8GB.

As you are on pt-1.7.1 it will always use the native amp, unless you specifically request to use `apex` with  `--fp16_backend apex`
 
> > Also you might want to try to disable `with autocast()`, perhaps you are hitting another caching bug - it will be slower since now fp16 will have to be reconverted many times, but you will be able to see if perhaps the memory overhead is due to caching still.

My apologies, this wasn't a good suggestion at all since it basically disabled the mixed-precision. I was trying to think how to disable just the caching to rule any leaks there, but I don't think there is a way. 

If you want to read about autocast caching, this comment from its creator is excellent:
https://discuss.pytorch.org/t/autocast-and-torch-no-grad-unexpected-behaviour/93475/3

> On my last (optional) bit in the original post -- just as a sanity check, any sense of whether the memory allocation vs sequence length looks as expected, or whether there should be much larger differences as sequence length increases/decreases? (I'm trying to get a sense of whether it's likely for me to fit T5-11b in these 4x40gb cards in the near term, or whether I'll need to wait for the full model parallelism/fp16/deep speed offloading, but these new models are new territory for me)

I'd have been nice to know when either DeepSpeed or fairscale get a chance to release ZeRO stage 3 support, but I'm not sure how to find it out - perhaps ask them if they have some possible projections? That would be the most desired news since then you will probably be able to fit a 45GB model over 4x40GB.

I think it'd be great to calculate all the different components and their memory requirements - then we can do the math easier. That is calculating how many bytes each component takes and how many of those we need.

Otherwise, I hope to have some Pipeline Parallelism working soon and perhaps we could try it on your 4x rig. 

Also has anybody attempted to distill t5-11b? If you could shave off some weight from it w/o losing much quality, perhaps it'd have been much easier to fit.
<|||||>> I was thinking we should start a new doc performance.md (markdown pretty please) where we discuss each of these issues. And you have just written a perfect entry on fp16.

I agree, and there is the table in the text-classification example that summarizes the speed gains. I have no time to do this this week, so if you want to go ahead and start a PR, feel free to do so!<|||||>Done: https://github.com/huggingface/transformers/issues/9824<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.