Related to abstractive text summarization
I was wondering if we could use XLM or XLM-R for abstractive text summarization.
If you are looking for summerization in non-English languages you can try using `MBartForConditionalGeneration`, or multilingual Bert using the `EncoderDecoder` framework. Not sure if xlm-r is yet supported in `EncoderDecoder`<|||||>Looking at xlm-r source code it seems that it can be easily added in EncoderDecoder as it subclasses Roberta which is supported in EncoderDecoder<|||||>Alright. I can try mBART and mBERT. 
What I was wondering about XLM was, if we could use it in a language modeling setting for this task, like how we use GPT for any seq2seq task.

Sending both text and the summary together and calculating the loss only over the summaries.<|||||>Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.<|||||>EncoderDecoder class allows you to use encoder only models as both encoder and decoder and fine-tune for seq-2-seq task. Here's an example of Roberta2Roberta fine-tuned on CNN dm https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16<|||||>Makes sense. Saw XLMWithLMHead in https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py, so just got curious.

> Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.

<|||||>> EncoderDecoder class allows you to use encoder only models as both encoder and decoder and fine-tune for seq-2-seq task. Here's an example of Roberta2Roberta fine-tuned on CNN dm https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16

Oh thank you so much!<|||||>Also what you said is doable, xlm-r can be used like a causal LM by configuring the attention mask. Might not give the best results though. See how RobertaForCausalLM is implemented. <|||||>> Makes sense. Saw XLMWithLMHead in https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py, so just got curious.
> 
> > Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.

Aah Sorry, typo, I meant XLM-R, not xlm<|||||>> Also what you said is doable, xlm-r can be used like a causal LM by configuring the attention mask. Might not give the best results though. See how RobertaForCausalLM is implemented.

Ohh, sure. Will check it out.<|||||>Also, what will be the best way to finetune T5  in a multi-task setting.<|||||>Also, are there any models we can use for code-switched data.<|||||>> Also, are there any models we can use for code-switched data.

Not too familiar with this, but seen few models on model hub and they used Bert.
https://huggingface.co/sagorsarker/codeswitch-hineng-ner-lince

> Also, what will be the best way to finetune T5 in a multi-task setting.

If you can cast all your tasks in text-2-text format then multi-task training can be done simply using task pre-fixes as shown in the paper. Also I think the performance will depend upon the tasks and datasets so some experimentation is necessary.  Most important thing when doing multi-task is how you sample examples from different tasks. See section 3.5.2 of T5 paper.

Also the best place to ask this question would be
https://discuss.huggingface.co/t/t5-finetuning-tips/684<|||||>Alright, thank you so much for the help !!<|||||>I tried using Xlmr2Xlmr but seems that regardless of what input I provide I get the same output; I checked to see the is_decoder flag is set to true in the decoder. This issue persists throughout the finetuning process<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
