Memory calculator for transformer models
### Feature request

This feature request is quite high level. 
The feature is some tool, function, object, etc. that takes in information such as model config, trainer arguments, and max sequence length and calculates the expected memory usage. It could for instance be used to produce a warning if expected memory exceeds current available memory, and lets you select your hyperparameters with memory usage taken into account (instead of trying out which params raises memory errors and not). I mentioned this on the HF discord and was encouraged to make a feature request

### Motivation

When working with this library and the trainer API,
I've been missing some type of tool that can calculate the expected memory consumption of your model training.
`RuntimeError: CUDA error: out of memory` haunts us all and can perhaps be better understood if we're able to pre compute expected memory to see if the error is expected or not. It also makes it easier to select hyperparameters with memory constraints.


### Your contribution

Should this be of interest:
* how it will be integrated should probably be agreed upon first. 
* I'm willing to contribute to this in the summer if nobody has picked it up by then, should my help be wanted
Reminds me of your current work on accelerate @muellerzr!<|||||>On my to do (asap) is to integrate that work from Accelerate. Should be in the next few weeks here. Basically how that one works is we retry the training loop reducing the batch size until we escape the CUDA OOM (this was request *many* times in our internal slack to integrate it here as well). Said implementation: https://github.com/huggingface/accelerate/blob/main/src/accelerate/memory_utils.py<|||||>Very interesting, I wasn't aware of this. Looking forward to the integration
Any thoughts on pre-calculating expected memory usage?
Or any reason why this would be unfeasible or impractical?<|||||>Without lots of very specific code, currently it is unfeasible (though won't be soon!).
The key is with pytorch's `meta` device. It currently doesn't work on all ops, but once it does we should be able to track all the sizes of the intermediate activation without real memory usage, getting us there. 

Otherwise we currently *could* by just doing the size of the model * the right number based on the optimizer selected, but we'd still be missing all of those intermediate activation sizes.

For now, the bs reducer is a good way to only add a few minutes (if not seconds) to get it going, hence why I went with that approach. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.