Try to play around with some hyperparameters (e.g. learning rate, other optimizer) and see if the behavior improves. Also, it might be helpful to use a small subset of your data (e.g. just 10 pairs) and try to overfit your model on it. If your model cannot overfit this small data sample, you might have some bugs in your code or the architecture is just not suitable for the use case.