Trainer evaluation doesn't return eval loss for question-answering.
As mentioned in this issue:
"""
Just a note that I tried `python run_squad_trainer.py --model_name_or_path bert-base-uncased --model_type bert --data_dir squad --output_dir /tmp/debug_squad/ --overwrite_output_dir --do_train --do_eval --evaluate_during_training --logging_steps 100`.

For some reason I don't get any evaluation metric during training (I was expecting `loss` or `eval_loss`).

_Originally posted by @borisdayma in https://github.com/huggingface/transformers/pull/4829#issuecomment-644338712_
"""
I'm facing the same problem. I'm trying to train with Trainer class over a QA dataset different from SQUAD. Everything works fine, the model learns based on train loss. However, I haven't been able to get the eval loss. I hope these pieces of code show how I'm configuring Trainer. Can somebody tell me if I'm doing something wrong?
```
training_args = TrainingArguments(
    output_dir="./models/prueba_2",  
    per_gpu_train_batch_size=16,
    per_gpu_eval_batch_size=32,  
    num_train_epochs=10, 
    logging_steps=10,  
    save_steps=25, 
    do_eval=True,
    evaluate_during_training=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=collator,
    prediction_loss_only=True,
    compute_metrics = EvalPrediction
)
```
I also tried without EvalPrediction in compute_metrics.
Thanks in advance !

I solved it editing trainer.py, in this line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L765 . I added start_positions and end_positions to that "possible labels names" list, and it worked. Review the bug, please.<|||||>Thanks @alexvaca0 !

cc @julien-c , @sgugger <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> I solved it editing trainer.py, in this line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L765 . I added start_positions and end_positions to that "possible labels names" list, and it worked. Review the bug, please.

I am having the same issue. Can you elaborate on how you solved this problem? I could not locate a possible label list and start/end positions?<|||||>#7191 should solve the problem described here.<|||||>@sgugger How can we calculate validation and evaluation loss for question answering finetuning pipeline (run_qa.py)