Hi
How can I pass any parameter values in search space such as Number of attention heads, vocab size etc. with Raytune.
I have implemented it with optuna but not getting anything woth Ray Tune. Please help me to get it done i Ray tune as backend in Transformer Trainer API for Hyperparameter serch.
def model_init(trial):
if trial is not None:
    num_attention_heads=trial.suggest_int("num_attention_heads", 1, 2 )
    vocab_size=trial.suggest_int("vocab_size", 1000, 1500)
else:
    num_attention_heads = 6
    vocab_size=1000

config = RobertaConfig(
    vocab_size=vocab_size, #output vocabulary size 
    max_position_embeddings=514, #position embedding 
    num_attention_heads=num_attention_heads, #number of possible attentions 
    num_hidden_layers=3,
    type_vocab_size=1,
    )

#basic model skeleton is RobertA with new configuration
model =  RobertaForMaskedLM(config=config)  
print("Model Parameters {}".format(model.num_parameters()))
return model

but how i can achieve same with Ray Tune as backeend