@BramVanroy I think one interesting question is that, for instance with support vector machines in a text classification task we have clear hyperparameters C  or gamma (the total budget for misclassified points and the parameter of the radial kernel) to tune.
For transformers, this seems less clear. It is not obvious which hyperparameters one can tune to improve performance that is. Any ideas?