@sgugger thanks for the ping 
@phosseini unfortunately you can’t do a sample without replacement with Ray Tune right now, but you can however easily do a grid search:
    def tune_config_ray(trial):
        return {"learning_rate": tune.grid_search([5e-5, 4e-5, 3e-5, 2e-5]),
                "num_train_epochs": tune.choice([4]),
                "per_device_train_batch_size": tune.choice([16])
               }

    best_trial = trainer.hyperparameter_search(hp_space=tune_config_ray,
                                               backend='ray',
                                               direction='maximize',
                                               n_trials=1,  # you may need to set this to 1, or else you'll evaluate the grid search many times over.
                                               )