How to fine tune small dataset?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Most people test bert on large dataset, but when it comes to small dataset, I assume the fine tune process and batch size maybe different. Besides, the dataset domain is twitter domain, which is kind of different from BERT pretrained corpus.
Could anyone gives some suggestions on finetuning BERT on small twitter dataset? Thanks in advance for any help.
Usually, you should train more epochs. Optimal batch size from 8 to 16. I'm not sure but maybe learning rate should be lower in this case. You can try<|||||>Thanks for your suggestion. I will try that. Besides, do you think I should modify the warmup step or just set it as 10% of total step just like in the original paper? <|||||>Influence of warmup steps is not clear for me. it looks like it's not so important for final quality but can speed up training a bit<|||||>@avostryakov  Thanks for your help!<|||||>@avostryakov  I wonder if I could ask another question. I know in BERT, weight decay of some layer is set to be 0 while others are set to be 0.01. So how to set the weight decay for other layer like the linear layer after bert output in finetuning?  <|||||>First of all, I'm not sure that weight decay is really important. I tried 0.1 and 0.01 for RoBERTa without a difference in quality. But ok, you can look in run_glue.py for this code:

```
no_decay = ['bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': args.weight_decay},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
        ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)```

As you can see you can set any decay weight for any parameter or layer, you need to know it's name.<|||||>@avostryakov  Thanks very much, maybe the learning rate is more important than weight decay <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hey @RoderickGu, Do you have any intuition now? If so could you share them with public?  <|||||>@ereday  First I think you should follow the most methods in BERT fine-tune process, such as use adamw. Besides, you could use a small batch size. Since dataset is small, I also suggested you to run it several times for the best learning rate. Hope this helps.