Machine Learning, NLP, Bert, Albert, Transformers.
in hidden-layer size. Increasing it with values larger than 4096 degrades the model performance. Performance of ALBERT large (3-layer configuration from the diagram above) with the increase of hidden size. The hidden size of 4096 is the optimal value. Conclusion At first sight, ALBERT seems a