I have been working with torchtext word embeddings (ex. GloVe) and now I would like to experiment with sentence embeddings.
I am using FAIR LASER: https://pypi.org/project/laserembeddings/ 22
With Word embeddings, I would carry out the following:

Load Embedding Layer with Embedding.from_pretrained()
Feed to Linear Layer, LSTM and some other linear layers.

I have simply replaced the word embeddings with the FAIR sentence embeddings, but the network is not learning. Is this an incorrect way to approach this?