Time series are hard to process compared to other datasets. When converting the series into windows, you can keep the windows of the same sizes. If you are using tf.data.Dataset, you will set the drop_remainder=True to drop all windows not having the size you provided.
In addition to @aditya1601 ideas on improving the model, you can try to schedule the learning rate.
I had the same experiment, tweaked different hyperparameters, but what accounted for the increments in the metric (Mean absolute error) was using the Learning Rate Scheduler and keeping the model simple. It might not be the same there, but this is what I was able to capture.
If you find it hard to write a series processing function with tf.data.Dataset, you can refer to this 5 or this 8 (CC:@Laurence_Moroney).