It doesn’t change the fact that your GRU is not part of the training process, i.e., the parameters of the GRU never change. You only train your linear layers. As such, you could probably just average all word embeddings of a sentence to get the sentence embedding. The GRU is currently not giving you any additional benefit.
Just because the loss is going down doesn’t mean it’s all fine. Your network (containing only linear layers) is very likely pick up some pattern to train on to bring the loss down.