If you do swap out weights of a trained model you’d need to update the parameter list within the optimizer.
I’m not 100% about the use of the detach function perhaps you’d be better just copying it directly across with model.x_layer.weight = model.x_layer.weight.copy()?