Could you check, what torch.cuda.device_count() returns?
Are you seeing more than a single GPU and might accidentally use the wrong one?
I’m not familiar with Singularity containers, but you could give it a try.