A question about the weight decay
https://github.com/huggingface/transformers/blob/7729ef738161a0a182b172fcb7c351f6d2b9c50d/examples/run_squad.py#L90

Should this by `layer_norm.weight`? Even seems you are not using weight decay at all.