When training QA models, albert-xxlarge-v2 uses much more GPU mem than Bert-large
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,  
         When I used run_square.py to train QA model, I found that albert-xxlarge-v2 uses much more GPU mem than Bert-large. Specifically, when using Bert large, I can set `Max_sequence_ength = 512`, `bash_size = 12`. But when I use albert-xlarge-v2, I can only set `Max_sequence_length` = 512, `bash_size = 6`.  
In fact, the number of parameters of Albert XXLarge is much less than that of Bert large, and the size of model file is the same. Why does Albert-xxlarge occupy more GPU memory when training QA model? Is it caused by more head parameters?
![image](https://user-images.githubusercontent.com/7105813/70107805-8a4bbb80-1682-11ea-8b0f-14f30465ce0e.png)

![image](https://user-images.githubusercontent.com/7105813/70107779-786a1880-1682-11ea-9668-26de2d606d53.png)

The parameters of Albert XXLarge is much less than that of Bert large, because albert used shared parameters in all transformer layers. But it does not reduce computation，Albert xlarge is 1.5 times lower than bert large and Albert xxlarge is 3 times<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>ALBERT repeats the same parameters for each layer but increases each layer size, so even though it has fewer parameters than BERT, the memory needs are greater due to the much larger activations in each layer.