Thanks for looking @joeddav! I’ve been spending all day trying to figure this out and I think that the issue is somewhere between the encoder layer and the BART model itself. Looking at the source code it looks like calling forward pulls the input into an encoded representation and that is passed to the BART model. These encoder_hidden_states are disconnected from the main model when I backprop. I’m wondering if the huggingface implementation decouples it somewhere for performance reasons (that would make sense, you shouldn’t need to backprop this far normally when training).
Hopefully @patrickvonplaten can weigh in with some insight. If not, I’ll try pytorch forums and SO.