Increased Memory Consumption In Containers
### System Info

Transformers version: 4.16.2
Python version: 3.8
Pytorch version: 1.11.0
CUDA version: 11.7
Docker version: 20.10.17

### Reproduction

Hello!

We noticed an interesting issue.

Currently, we have a monolithic app with 2 PyTorch models - model A and model B(both GPT based models).

If we run the app with only Model A enabled - it consumes 2.5 GB GPU.
If we run the app with only Model B enabled - it consumes 2.2 GB GPU.
If we run the app with Model A and Model B together - memory consumption is less than model A + model B launched separately.

At the same time, if we divide the monolithic app into 2 smaller apps (Model A in container A, model B in container B) and run it via Docker, we see that GPU memory consumption is higher than if we run it as a monolithic single app. It literally becomes 4.7 GB (2.5 GB + 2.2 GB).

It seems that PyTorch reserves some GPU for itself. So, GPU memory consumption in a multi-services setup is higher than in a monolithic.
Is is a bug or is it a planned behaviour?

Could you, please, give us some hints if there are any ways to optimize memory consumption for PyTorch and if it is possible to share PyTorch reserved memory between different Docker containers?

Thanks!

### Expected behavior

Memory consumption in containers and in monolithic app should be the same
Hi @ai-john  👋 As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗

As you wrote, ML frameworks do allocate some GPU memory for themselves :) Hugging Face has deployment optimization solutions, see [Optimum](https://huggingface.co/docs/optimum/index)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.