I’m following the tutorial here:

  

      github.com
  

  
    huggingface/notebooks/blob/master/transformers_doc/training.ipynb 1


    {
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets\n",
    "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
    "# ! pip install git+https://github.com/huggingface/transformers.git\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning a pretrained model"
   ]



  This file has been truncated. show original

  

  
    
    
  

  


which creates two separate Trainers, one for training, the other for evaluation:

trainer = Trainer(
    model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.evaluate()

Three questions:

Why are two Trainers required? What can we just create the Trainer once, call .train() then call .evaluate()?
Why is the validation loss not computed during training?
Why is a custom compute_metrics function required? I’m referring to:


metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)