In the PyTorch Seq2Seq tutorial 17, we see that the decoder’s embedding layer goes through a relu before feeding it to the GRU but for the encoder there isn’t the relu layer.
Why is that so?
What is the purpose of the relu layer in the decoder after the embedding? Why didn’t the encoder’s embedding need a relu or any other non-linearity before feeding it to the GRU?