Solved by sgugger in post #4 
              
              
                You can see the batches that will be passed to your model for evaluation with: 
for batch in trainer.get_eval_dataloader(eval_dataset):
    break

And see if it does contain the "labels" key.