Ah yes you’re right, it is indeed in the wrong place and should be put first, thanks for the lengthy explanation, I was a bit too tired when I read your post first 
However there is a line # Add weight decay at the end (fixed version) in the code introduced by @thomwolf so there is probably a reason this is put in second.