If a single sample is raising the out of memory issue, your GPU memory capacity might just be too small for this workload.
If you have another workstation with a bigger GPU, you could try to run it there and check the memory usage for a single sample.
Alternatively, you could also try to run it on Colab, as you might also get a GPU with more than 4GB of memory.