How many samples are in your dataset?
Have you tried using torch.vision transforms to augment the image data during training(shut off during eval)?
During eval, are you using:

with torch.no_grad():
    model.eval()
    ... # eval steps
model.train()

Sounds like an issue of too little data and no transforms.