[bart] decoder.last_hidden_state shape changes when passing labels
```
config = BartConfig(
            vocab_size=99,
            d_model=24,
            encoder_layers=2,
            decoder_layers=2,
            encoder_attention_heads=2,
            decoder_attention_heads=2,
            encoder_ffn_dim=32,
            decoder_ffn_dim=32,
            max_position_embeddings=48,
            add_final_layer_norm=True,
        )
        lm_model = BartForConditionalGeneration(config).to(torch_device)
        context = torch.Tensor([[71, 82, 18, 33, 46, 91, 2], [68, 34, 26, 58, 30, 2, 1]]).long().to(torch_device)
        summary = torch.Tensor([[82, 71, 82, 18, 2], [58, 68, 2, 1, 1]]).long().to(torch_device)
        loss, logits, enc_features = lm_model(input_ids=context, decoder_input_ids=summary, labels=summary)

        expected_shape = (*summary.shape, config.vocab_size)
        self.assertEqual(logits.shape, expected_shape)
        outputs2 = lm_model(input_ids=context, decoder_input_ids=summary)
        self.assertEqual(outputs2.logits.shape, expected_shape) 
# Fails torch.Size([2, 1, 99]) != (2, 5, 99)
```
Is this expected @sgugger ?
NVM, it's a me problem.<|||||>FFR, You have to manually pass `use_cache=False` to bart forward now if you're not generating