I’d say it’s somewhat expected that it doesn’t learn, because the pre-trained model expects some input and the input it gets is completely different.
I’d maybe initialize the weights of the new_layer such that they start as the identity (with zero bias), so that your network has at least some chance to optimize.
Also, I’d advise using self.pretrained_model((x, (hx, cx))) instead of calling forward, but that’s a detail.