Hi all,
I have been working on implementing an encoder-decoder NLP model and recently I switched from using learnable embeddings with a vocabulary of ~20,000 (all the words in my dataset) to using GloVe pre-trained embeddings with vocabulary size ~400,000 (I freeze the embedding layer).
This change has been problematic as with the larger vocabulary size now the following tensors become much larger, especially the ones in bold:

Embedding Layer (vocab_size, embed_size)
Decoder fully connected layer weights (hiddenstate_dim, vocab_size)
This layer transforms the hidden state of the LSTM to logits over the whole vocabulary, then they are softmaxed to get a probability distribution.
Decoder output after fully connected layer (batch_size, max_seq_length, vocab_size)
Decoder output after softmax (batch_size, max_seq_length, vocab_size)

Tensors created during the backward pass
I checked the cuda memory allocated and the memory change before and after loss.backward() is very large.

This means that during the forward and backward passes, memory allocations are too high to have large batch sizes (100-300), I can only go up to 15-20 and can’t train efficiently anymore.
My questions are:

Is this something that makes sense and that is to be expected with large embeddings?
Am I thinking of using pre-trained embeddings right? As in, my training dataset corpus has vocab size 20,000 but the GloVe pre-trained embeddings I intend to use have 400,000 tokens. What is the standard way when using pre-trained embeddings? To use the entire weight matrix or to still only use the 20,000 embedding layer with only the words in my dataset corpus but initialise these with the relevant pre-trained GloVe-embeddings?

By the way, when using the pre-trained embeddings, I freeze the embedding layer so sparse=True does not solve anything.
Thank you!