Hi, I am using a simple RNN/GRU for my problem.
My inputs are continuous and  on a scale of 0-1000 and variables to be predicted are continuous as well and on a scale of 0-800.
I’ve observed that with different random seeds, set using: torch.manual_seed(seed), I get very different training performance, all other things kept constant (learning rate, num_layers, hidden_size, bidirectional). For some seeds, I don’t see the loss changing at all. While for other seeds, the loss reducing with iterations as expected.
The prediction I would get would be all 0s. Now, it should be noted that I have the following two lines in the forward, which one might not have in a vanilla RNN. These are dictated by the model.
# Prediction should be non-negative
pred = torch.clamp(pred, min=0.)
# Prediction should always be lesser than or equal to the input
pred = torch.min(pred, x)

I was wondering why we’d have the case for the loss not changing at all, and why would it be so sensitive to
random seed. I have a hunch that it’s related to the 0 clamping that I’m doing, but I don’t know how to avoid that!
Here’s the RNN block I’m using:
class CustomRNN(nn.Module):
    def __init__(self, cell_type, hidden_size, num_layers, bidirectional):
        super(CustomRNN, self).__init__()
        torch.manual_seed(1)

        if bidirectional:
            self.num_directions = 2
        else:
            self.num_directions = 1
        if cell_type=="RNN":
            self.rnn = nn.RNN(input_size=1, hidden_size=hidden_size,
                   num_layers=num_layers, batch_first=True,
                   bidirectional=bidirectional)
        elif cell_type=="GRU":
            self.rnn = nn.GRU(input_size=1, hidden_size=hidden_size,
                              num_layers=num_layers, batch_first=True,
                              bidirectional=bidirectional)
        else:
            self.rnn = nn.LSTM(input_size=1, hidden_size=hidden_size,
                              num_layers=num_layers, batch_first=True,
                              bidirectional=bidirectional)

        self.linear = nn.Linear(hidden_size*self.num_directions, 1 )

    def forward(self, x):
        pred, hidden = self.rnn(x, None)
        pred = self.linear(pred).view(pred.data.shape[0], -1, 1)
        pred = torch.clamp(pred, min=0.)
        pred = torch.min(pred, x)
        return pred