Updating simple_lm_finetuning.py for FP16 training
in simple_lm_finetuning the recent updated code doesn't work with the old optimizer specifications. 

When not running with --fp16 
`
optimizer = BertAdam(optimizer_grouped_parameters,
                     lr=args.learning_rate,
                     warmup=args.warmup_proportion,
                     t_total=num_train_optimization_steps)
// In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=args.learning_rate, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLi
`

fixes the problem as suggested.

But when running --fp16 
`scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_train_optimization_steps)  # PyTorch scheduler`
will error out with FP16_Optimizer saying that 

> TypeError: FP16_Optimizer is not an Optimizer

Does the FuseAdam object need to be passed into WarmupLinearSchedule instead?
Hi, has this been fixed? I've tried updating my language modeling script to match but still getting errors.<|||||>Having the same problem at the moment ...<|||||>I guess the preferred way is to use  `apex.amp` like in this example?
https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
