you’ll need to set up the evaluation dataset / dataloader and write your own evaluation loop to compute the metrics.
you can find examples of this in the examples/legacy directory of the transformers repo, e.g. here is the evaluation for question-answering on squad: transformers/run_squad.py at 6e1ee47b361f9dc1b6da0104d77b38297042efae · huggingface/transformers · GitHub 64