I am wondering whether this is still correct. So what you do is, for all input sequences:
neg_log_likelihood = outputs[0] * trg_len

Yet the first output of causal LMs is CrossEntropyLoss, not NLLL. So from that you can just get the mean CE loss from all sequences and get the exponential.
EDIT: that is also how it is implemented in the Trainer and run_clm.py script. First gather all losses for  all batches in the whole validation set and take the mean.

  

      github.com
  

  
    huggingface/transformers/blob/11c69b80452fae4b13c6d8bc22bdc19f3a752199/src/transformers/trainer.py#L2353-L2354 2



      
      
          if all_losses is not None:
              metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item()
      
    
  


  

  
    
    
  

  


Then take the exponential.

  

      github.com
  

  
    huggingface/transformers/blob/11c69b80452fae4b13c6d8bc22bdc19f3a752199/examples/pytorch/language-modeling/run_clm.py#L495 3



      
      
          
          
# Evaluation
          if training_args.do_eval:
              logger.info("*** Evaluate ***")
          
          
    metrics = trainer.evaluate()
          
          
    max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
              metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
              try:
                  perplexity = math.exp(metrics["eval_loss"])
              except OverflowError:
                  perplexity = float("inf")
              metrics["perplexity"] = perplexity
          
          
    trainer.log_metrics("eval", metrics)
              trainer.save_metrics("eval", metrics)
          
          
kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-generation"}
          if data_args.dataset_name is not None:
              kwargs["dataset_tags"] = data_args.dataset_name