I’m not sure I understand your forward method completely, but it looks like you are just using weights if pretrained is set to True. The output of your encoder won’t be used anymore, which might explain the missing gradients.