I haven’t checked the code, I’m on mobile now. But are there many scenarios where we actually need to do hyperparameters search on the evaluation/inference side?  In addition, does this use the optuna implementation that is being worked on in the trainer by @sgugger , or is it a separate implementation?