I tried to train a model with batchnorm layers. During the training, I set model.train(). Every 100 iteration, I validate the accuracy and set model.eval(). However, the validation is not correct. I don’t think this is due to overfitting because even if I use the same image as training, the testing loss is also quite different from the training loss. Also, if I still set model.train() during testing, the testing loss is correct. But such usage does not make sense because my model contains batchnorm layers.
Below is my training code:
for epoch in range(num_epochs):
    epoch_loss = 0.0
    optimizer = lr_scheduler(optimizer, epoch)

    for iteration, data in enumerate(dataloader, 0):
        iter_index += 1
        label_patch = data['label_patch']
        residue_patch = data['residue_patch']
        stacked_patch = data['stacked_patch']
        microshift_patch = data['train_patch']
        inputs, residues, labels, microshifts = Variable(stacked_patch.type(dtype)), Variable(residue_patch.type(dtype), requires_grad=False), Variable(label_patch.type(dtype), requires_grad=False), Variable(microshift_patch.type(dtype), requires_grad=False)

        # set model to train mode (before zero grad)
        model.train()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward
        outputs = model(inputs)
        loss = criterion(outputs, residues)

        # backward + optimize only if in training phase
        loss.backward()
        optimizer.step()

        # statistics
        epoch_loss += loss.data[0]

        # test the model every 100 iterations
        if iter_index % logging_iter == 0:
            loss_test, psnr_test = test_model(model)  

    # checkpoint for each epoch
    model_out_path = "checkpoints/model_epoch_{}_residue.pth".format(epoch)
    torch.save(model, model_out_path)