I have a problem very similar to A strange problem about the gpu processes and num_workers 3
In particular the following happens and I don’t understand why there are extra processes per GPU that are consuming a lot of GPU ram:

image556×802 23.4 KB

Could you please advise what could be the problem here?
I’m using PyTorch-Lightning for training using multiple GPUs. I have 4 V100 GPUs and the following dependencies:

torch==1.7.1
torchvision==0.8.2
CUDA 11.0