Thanks for replying. 
Is there an example about using ray tune with huggingface Trainer to do hyperparameter searching for dropout rate? I’m not sure about how to set the ray tune trial object properly in the model_init. 
I find lots of material about searching for learning rate, batch size and etc. None of them are related to searching for different dropout rates. The documentation said model_init together with the trial object should be able to handle this…