Machine Learning, NLP, Bert, Albert, Transformers.
weights belong to a single transformer block. The weights are then shared for each of the 12 layers. Therefore, during training or inference, the algorithm has to be executed on more than 2 billion parameters! Due to these reasons, ALBERT is suited better for problems when the speed can be traded