I was trying to train a neural network that uses ResNet 152 as backbone but I was getting CUDA out of memory error. After that, I added the code fragment below to enable PyTorch to use more memory.
torch.cuda.empty_cache()
torch.cuda.set_per_process_memory_fraction(1., 0)

However, I am still not able to train my model despite the fact that PyTorch uses 6.06 GB of memory and tries to allocate 58.00 MiB where initally there are 7+ GB of memory unused in my GPU.
RuntimeError: CUDA out of memory. 
Tried to allocate 58.00 MiB (GPU 0; 7.80 GiB total capacity; 6.05 GiB already allocated; 
48.94 MiB free; 7.80 GiB allowed; 6.19 GiB reserved in total by PyTorch)

The result that I obtain after running nvidia-smi command.

| N/A   47C    P8     6W /  N/A |    362MiB /  7982MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       947      G   /usr/lib/xorg/Xorg                 70MiB |
|    0   N/A  N/A      1549      G   /usr/lib/xorg/Xorg                159MiB |
|    0   N/A  N/A      1722      G   /usr/bin/gnome-shell               34MiB |
|    0   N/A  N/A      6506      G   ...AAAAAAAAA= --shared-files       85MiB |
+-----------------------------------------------------------------------------+

How can I increase the 6.19 GiB memory reserved in total by PyTorch to use more memory
from my GPU? Thank you!
OS: Ubuntu 20.04
GPU: Nvidia GeForce RTX 2070-SUPER Max Q Super Design
PyTorch version: 1.8.1+cu111
Cuda toolkit: 11.2
Nvidia Cuda driver: 460.80