You could create unit tests by storing the values of your parameters and check for updates after a training iteration. The concept is explained here 1.8k. It’s for Tensorflow, but you can easily adapt it to PyTorch.