As the error message claims, you are running out of memory and would need to reduce the memory usage in your script by e.g. reducing the batch size or using torch.utils.checkpoint to trade compute for memory.
The not reported allocated memory would be used by the CUDA context. If nvidia-smi doesn’t report any processes, it might be a permission issue, but the GPU usage should nevertheless be shown.