Maybe I misunderstand something wrong, but I think nothing should change when using pretrained word embeddings. When your dataset has a vocabulary of size 20k, you only need the 20k respective word embeddings.
An embedding layer is essentially just a look-up layer. So there is no need for embeddings that are never looked up :). Just create a embedding matrix of size (vocab_size, embed_dim) – e.g., (20000, 300) – and fill the 20k embeddings using the pretrained ones.