I am trying to rebuild a Keras architecture in pytorch, which looks like this
    rnn_layer1 = GRU(25) (emb_seq_title_description)
    # [...]
    main_l = Dropout(0.1)(Dense(512,activation='relu') (main_l))
    main_l = Dropout(0.1)(Dense(64,activation='relu') (main_l))
    
    #output
    output = Dense(1,activation="sigmoid") (main_l)

So I tried to adjust the basic RNN example in pytorch and add ReLUs to the Linear layers.  However, I am not sure if I can call ReLU directly in the forward method or should call it in the init method.
My first try looks like this:
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)


    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = nn.ReLU(self.i2h(combined))
        output = nn.ReLU(self.i2o(combined))
        output = nn.sigmoid(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

n_hidden = 128
n_letters = 26
n_categories = 2
rnn = RNN(n_letters, n_hidden, n_categories)

However, when I look at the rnn object in python, I will not see the ReLUs, so maybe it’s not right to call nn.ReLU directly in the forward method…
RNN(
  (i2h): Linear(in_features=154, out_features=128, bias=True)
  (i2o): Linear(in_features=154, out_features=2, bias=True)
)