Hello @valhalla
Thanks for the reply. After wrestling with this for a little longer I realized that TransfoXLLMHeadModel's forward method isn’t compatible with the Trainer class because it returns a 1D Tensor of losses rather than a scalar loss. This is confirmed in this 1 issue I created on Github. I’m going to try and create a custom Trainer to reduce this loss and post my results here.