Machine Learning, NLP, Bert, Albert, Transformers.
vocabulary matrix V x H can be decomposed into a pair of smaller matrices of sizes V x E and E x H. Vocabulary matrix factorization As a consequence, instead of using O(V x H) parameters, decomposition results in only O(V x E + E x H) weights. Obviously, this method is effective when H >> E.