I’m not familiar with Lightning so pinging @williamFalcon in case he’s seen something similar.
Are you seeing the same issue using “plain PyTorch” and if so could you post a code snippet we could debug?