@FruitVinegar interesting situation.
What does evaluator behind is (link 3) :
model.eval()
with torch.no_grad():
    x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
    y_pred = model(x)
    # ...

Can you check if you also have NaNs with the following code :
logfile = open('logs/manual.log', 'w')

dataloader_iter = iter(dataloader)

model.eval()
with torch.no_grad():
    for i in range(500):
        x, t = next(dataloader_iter)
        y = model(x)
    
        logfile.write('{0}th iteration...\n'.format(i))
        logfile.write('    x: {0}\n'.format(x))
        logfile.write('    O: {0}\n'.format(y))
        logfile.write('    o: {0}\n'.format(torch.argmax(y, dim=1)))
        logfile.write('    t: {0}\n\n'.format(t))
    
logfile.close()

Can you detail how your model is defined such that I could reproduce the issue from my side ?