Why does setting `--fp16 True` not save memory
https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/examples/pytorch/language-modeling/run_mlm.py#L247-L250

https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/src/transformers/training_args.py#L663-L666

I want to pre-train Roberta on my dataset. However, the Batch size can be set to 32 at most. Otherwise, OOM is reported. I plan to use `Mixed-precision` to save memory. So I set `--fp16 True`. However, the Batch size can only be set to 32 at most. Otherwise, OOM will be reported. It seems that setting up `FP16` is not doing much to save memory.
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>This question is now on the forum here: https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400