Hey @gildesh I’m not sure why you say BART will provide extractive summaries - my understanding is that it is an encoder-decoder Transformer, so the decoder will generate summaries if trained to do so.
In any case, the reason why you get an error with predict_with_generate=False is because the Trainer won’t call the model’s generate() method in that case (it just computes the loss / logits, which is why you don’t see the metrics).
So if you want to compute things like ROUGE during training, you’ll need to generate the summaries with predict_with_generate=True
PS the notebook you shared looks more complicated than it needs to be. I recommend using the official summarization example as a foundation (which will certainly work with BART)