BertAdam gradient clipping is not global
Just took a look at the gradient clipping algorithm used in: https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ae8c8be1e3fc770968cd3fdb3b643e0b166e540/pytorch_pretrained_bert/optimization.py#L270

It's clipping gradients to a local norm of 1. It should be clipping gradients to a global norm of 1 as in https://github.com/google-research/bert/blob/master/optimization.py#L74 or  in https://github.com/NVIDIA/Megatron-LM/blob/master/pretrain_bert.py#L226 .
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
