The performance of the model will depend on the usability of the used pretrained embedding.
Assuming you are using a standard pretrained embedding matrix, your inputs might not be well represented and the embedding might need some fine tuning to achieve reasonable results.
Could you explain your use case a bit more, i.e. which embedding are you using and what do your input tensors represent?