I also found this statement in the Transformers introductory page on Summary of the tasks 1:

Note
All tasks presented here leverage pre-trained checkpoints that were fine-tuned on specific tasks. Loading a checkpoint that was not fine-tuned on a specific task would load only the base transformer layers and not the additional head that is used for the task, initializing the weights of that head randomly.
This would produce random output.