I’d agree with @BramVanroy , any thoughts @joeddav on the above post?



 BramVanroy:

Yet the first output of causal LMs is CrossEntropyLoss , not NLLL. So from that you can just get the mean CE loss from all sequences and get the exponential.


I don’t understand the multiplication by trg_len in this example. Also on my dataset it explodes the perplexity by orders of magnitude above a uniform upper bound of log(|Vocab Size|)