Hello @infinitejoy.



 infinitejoy:

I dont think its the exact same model


How the model could not be the same?
When you run trainer.evaluate(), the model used is this one: trainer.model.
Just run trainer.evaluate() 2 times in a row in the Colab notebook language.modeling.ipynb 1 and you’ll see a different perplexity (… with the same model).
import math

eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")




 infinitejoy:

I dont think its (…) the same evaluation dataset


Same comment: when you run 2 times the same code trainer.evaluate(), it is the same evaluation set defined at the beginning of the notebook by the following code:
from datasets import load_dataset
datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')

Could you run the Colab notebook and publish your results? Thanks. (I did it for Masked LM, not for Causal LM)