can you take out of evaluation the work with threads?
torch.set_num_threads(1)
you are not working with threads at train.
I would try making the evaluation the exact same as train, just adding with torch no grad to make sure this is in eval mode