Machine Learning, NLP, Bert, Albert, Transformers.
increase of number of layers. Models in the diagram with ≥ 3 layers were fine-tuned from the checkpoint of the previous model. It can be observed that after reaching 12 layers, the performance increase gets slower and gradually falls after 24 layers. A similar phenomenon occurs with the increase of