Looks like the problem actually was the optimizers.  One other thing I was doing is reloading the model.  I think with an old verion of pytorch the optimizers would automatically point to the reloaded layers, but now it appears to have to be reinitialized.  Looks like that fixed it!