You call loss.backward() when you need to backpropogate your loss to your weights, which happens during training, you need not do it in validation step. You can load the weights to the model from the checkpoints and compute the loss from the output of the model directly.