I guess this is more of a generic answer then specific to PyTorch, but two things come to mind:


If you reuse a pre-trained network, ensure that when you train on your images you “freeze” most of the earlier layers. So only train the last few layers (typically the fully connected layers) and don’t update the weights of the earlier layers. Possible you are already doing this, but if not I would suggest to try this first.


I like to use dropouts as a way to prevent overfitting. The default dropout value of 0.5 is in my personal experience a good starting point. You could add them at the end between the fully connected layers.