Well, this happens whether I use an sbatch script or do an interactive session with srun. Both times I request 1 or 2 GPUs with --gres=gpu:n. There isn’t really room for something to go wrong with the script so I find it hard to imagine that could be the problem. It’s possible it’s an issue with pytorch-lightning but the problem only happens with my pre-trained Inception model which is just a nn.Module.