It seems your model is not learning the right pattern.
I would recommend to select a small subset of the data (e.g. just 10 samples) and try to overfit this dataset.
Once your model successfully predicts the target, you could then scale up the use case.
If your model is unable to do so, there might be a bug in the code or the model architecture is not suitable for this task.