You can adjust the weight_decay in your TrainingArguments. For the dropout in the model, you can adjust it by passing it to you XxxModel.from_pretrained call. The exact name of the argument varies depending on the model, so you should check the documentation of the config of the model you are using.