Is this issue still not resolved! Sad.
I too am facing same problem.

RuntimeError: CUDA out of memory. Tried to allocate 540.00 MiB (GPU 0; 4.00 GiB total capacity; 1.94 GiB already allocated; 267.70 MiB free; 2.10 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Only half of memory(1.94GB) is being used, rest half (2.10GB) is not being used, Or I’d say is being wasted!!