I’m not quite sure what the sequence dimension here is (if you use LSTMs). It would seem you are feeding 1-element sequences? In that case, Linear might be what you need, but I feel I’m missing something.
The generic recommendation in a situation like this is to try to feed in the same batch over and over again and see if the loss decreases (it should), i.e. to see if your network can overfit.
Best regards
Thomas