Machine Learning, NLP, Bert, Albert, Transformers.
showed that ALBERT models with wide hidden sizes (≥ 1024) do not benefit a lot from an increase in the number of layers. That is one of the reasons why the number of layers was reduced from 24 in ALBERT large to 12 in the xxlarge version. Performance of ALBERT large (18M parameters) with the