Hey! I think you are thinking about this the right way. The dangers of saving the model on the best training loss is that you risk overfitting. It would be better to save when your loss is low on a validation set and I’d say that 1 image is probably not enough - but you might not need that many if you do random crop on a few big validation images.
Yes, you should use with torch.no_grad() when you don’t train to speed things up. Could also be nice to put the model in eval() mode.
I don’t know why your “epoch_loss kept jumping in a large scale” - it wasn’t just a fluke?
PS: You can save several models if you have a decently sized hard drive. Like the best validation loss, training loss, validation accuracy. Just need to save them at different paths