I see. In the model, enc flows through the first part of the network that enc_pred does. Could this be an issue?
For example, does autograd update the network by flowing through enc, then throw an error when flowing through enc_pred since weights have been modified in place?