Machine Learning, NLP, Bert, Albert, Transformers.
take the BERT base model: it has a vocabulary of 30K tokens, each represented by a 768-component embedding. In total, this results in 23M weights to be stored and trained. For larger models, this number is even larger. This problem can be avoided by using matrix factorization. The original