Make sure that training and testing procedure is as follows:

model = my_model()

for epoch in Epochs:
  model.train()
  for train_batch in Batches:
            ######Train
  model.eval()
  for test_batch in batches:
            ######Test