Should we choose neural network with highest performance (F1/Accuracy) on the validation set, or with lowest validation loss? I am doing a text classification task through fine-tuning pre-trained language models + classification head.
I would like to know whether I should choose the neural network with the highest performance on the validation set, or the network with the lowest loss on the validation loss.

I'd probably go for the Brier score (the mean squared error on the labels) rather than the cross-entropy loss as the cross-entropy loss is likely to be dominated by very confident miss-classifications, which are likely to be far from the decision boundary (so some proper scoring rules are better than others - you want one that gives relatively more attention to the region near the decision boundary).  The validation set is usually rather small, so the accuracy is likely to be brittle (unreliable) and minimising the Brier score is likely to give better accuracy on the test set (rather than the validation set).
If the validation set is large enough, then selecting based on the accuracy is likely to be better, IF accuracy is the most important metric for the needs of your application.
I've found this approach is very effective for tuning the hyper-parameters of kernel learning methods (www, pdf)
