Perplexity in T5/BART
Hi,
Is there a way to compute perplexity for T5 or BART outputs?
T5 uses CrossEntropy loss so I think you can use `torch.exp(loss)` to calculate perplexity.<|||||>Thanks!<|||||>@patil-suraj Sorry to bother you again, I thought my problem has been solved, but I don't know how to deal with the loss. It's a (1,1,vocab_size) tensor. I don't even have an intuition about what that means. Can you help me?<|||||>It seems that you are not passing `lm_labels`. When you pass `lm_labels` to T5/BART's forward methods then the first element of the output tuple is the loss, otherwise its `prediction_scores` tensor which you are receiving.<|||||>hi @patil-suraj bart doesnt have lm_labels has argument<|||||>`lm_labels` is renamed  to `labels`<|||||>Thanks for the prompt response, @patil-suraj , previously before trying Bart, I trained it with t5,the valid loss gets to almost zero, but still found it difficult generating correct translation. I would love to know reasons for that. Thanks<|||||>I'm not sure which problem you are talking about, hard to answer without knowing the context.<|||||>Hi patil,

I am working on a translation task/competition, so as a text2text
framework, I used t5 even though it is not pretrained on my source language
just for the knowledge sake,
The notebook link here

https://colab.research.google.com/drive/1w0brh9KULV4zDmgMme4csXeCpfAR960N?usp=sharing

the test and valid loss was declining to zero even though the later
generated predictions wasn't good so I changed to Bart even though no
improvement, I have no idea why? I would love to know the reason.

I found MarianMT just yesterday which was pretrained for my task yo-en, I
have started working with it. I think it would also be the case for MarianMt
Thank you.

On Fri, Jan 1, 2021, 8:38 AM Suraj Patil <notifications@github.com> wrote:

> I'm not sure which problem you are talking about, hard to answer without
> knowing the context.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4210#issuecomment-753280527>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOP3ZJFLJLLFGUSFCU3WKB3SXV3X7ANCNFSM4M3PUPXA>
> .
>
