This might be related, as it discusses a recently fixed bug with a Colan notebook.

  
    
    
    Using hyperparameter-search in Trainer 🤗Transformers
  
  
    Hi @sgugger, in case you’re not aware of it, it seems the latest commit on master broke the Colab notebook you shared on Twitter 
Trying to run that notebook, I hit the following error when trying to run 
best_run = trainer.hyperparameter_search(n_trials=10, direction="maximize")

with the optuna backend. 
Stack trace: 
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:900: RuntimeWarning:

invalid value encountered in double_scalars

[W 2020-10-22 14:58:41,815] Trial 0 f…
  


But to answer your question concerning the results: the results when using the Trainer or your own trainer should be the same as long as you use the same loss function and hyperparameters.