Issue with lr during training from scratch
I'm trying to resume my pretraining of RoBERTa model and I noticed that everything is OK except learning rate. I thought that it should continue to decrease from last checkpoint (lr = 7e-7) but lr is much bigger (lr = 5e-5) than it has to be.

this is my parameters to run script:

- 	--model_name_or_path ./ROBERTA-small-v1/checkpoint-30000
-   --train_data_file TEXT.txt
-   --output_dir ./ROBERTA-small-v2              
- 	--mlm
- 	--config_name ./ROBERTA_SMILES-v1/checkpoint-30000
- 	--tokenizer_name ./ROBERTA
- 	--do_train
- 	--line_by_line
- 	--num_train_epochs 10
- 	--save_total_limit 1
- 	--save_steps 2000
- 	--per_gpu_train_batch_size 16
- 	--seed 42
- 	--logging_steps 250

Are you restarting from your checkpoint with the exact same set of script parameters?<|||||>@julien-c  Actually, no. And I guess I undestood what the reason of that behavior is. Originally I used to train with number of epochs = 5. After that I selected 10 epochs and hoped that it'll continue pretraining from 5'th epoch up to 10'th. However as far as I understand lr-value is calculated based on the number of epochs. The current value of lr is the same as I got at 15k iteration (30k/2). So I have should selected 10 (or whatever number) epochs originally<|||||>Yes. You can always implement this behaviour yourself if you need it.