Training beyond specified 't_total' steps with schedule 'warmup_linear'. Learning rate set to 0.0. Please set 't_total' of BertAdam correctly.
I am seeing the above error in my training process. Is it a significant issue? Looks like it's related to `t_total`, which should be properly set here:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/b832d5bb8a6dfc5965015b828e577677eace601e/examples/run_classifier.py#L742-L743

What could be potential causes of this issue? I trained exactly `args.num_train_epochs` epochs, and didn't alter the training data in between, so shouldn't this pre-calculated `t_total` work without issue?

My `len(train_examples)` is 49401, `args.num_train_epochs` is 5, using 2 GPUs, and other parameters are left as default. If it matters, my code is based on a version (68a889) before the recent `WarmupLinearSchedule` change.
Actually, shouldn't the `int()` be a `ceiling()`? Because let's say `args.gradient_accumulation_steps` is 1, then it is `ceiling(len(train_examples) / args.train_batch_size)` that is the number of batches in an epoch.<|||||>I am having the same problem with my finetuned model for gpt2<|||||>I am having the same issue in partially changed run_squad.py code.<|||||>I have the same issue.<|||||>Humm yes, we should probably change `int` to `ceiling` in this example indeed.<|||||>Is this a significant issue? If it's only the last few batches in the last epoch that are not being trained on, it shouldn't be a huge problem, right?

Also I find it strange that suddenly a lot of people are running into this bug in this past week (according to the replies to this issue) even though the `int` code was written 3 months ago. Is this also related to some other more recent changes?<|||||>Yes there was a huge refactoring of the `BertAdam` optimizer by @lukovnikov (#389, #445, #531)<|||||>Hi, this warning is printed to avoid wasted computations with warmup-linear or other surprises with other schedules due to a t_total set too low.
And I think that line should be `int( math.ceil(len(train_examples) / args.train_batch_size) / args.gradient_accumulation_steps) * args.num_train_epochs` (@thomwolf) ?<|||||>Hi. I figured out the source of the problem: t_total, aka num_train_optimization_steps 

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L903

 is computed over the length of the train examples, while the true number of steps is determined by whatever convert_examples_to_features returns

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L970

A print(len(train_examples), len(train_features)) in line 980 returns:

87599 191597 

<|||||>You could also add the option `drop_last=True` to the `DataLoader`, then the number of samples will be calculated correctly.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I get error below while running the program.. Did I do any mistake?

warmup_linear = WarmupLinearSchedule( warmup=args.warmup_proportion,
t_total=num_train_optimization_steps)

lr_this_step = args.learning_rate * warmup_linear.get_lr(num_train_optimization_steps,
args.warmup_proportion)

WARNING - pytorch_pretrained_bert.optimization - Training beyond specified 't_total'. Learning rate multiplier set to 0.0. Please set 't_total' of WarmupLinearSchedule correctly.