Could you explain, what dropout=0.2 and recurrent_dropout=0.2 do in your Keras model?
It seems that you are using a single F.dropout on the outputs of both LSTM layers.
Also, since you are using the functional dropout API, you should pass the training=self.training argument to disable it during validation.