I am a beginner of the PyTorch, now I am writing a code for the time series forecasting by LSTM. The LSTM includes two layers and stacks together. The first layer of LSTM includes 10 LSTM units and the hidden units will pass to another layer of LSTM which includes a single LSTM unit. The following is the architecture diagram of the neural network.

following is my code,
def __init__(self, nb_features=1, hidden_size_1=100, hidden_size_2=100, nb_layers_1 =5, nb_layers_2 = 1, dropout=0.4): #(self, nb_features=1, hidden_size=100, nb_layers=10, dropout=0.5):      ####### nb_layers=5
    super(Sequence, self).__init__()
    self.nb_features = nb_features
    self.hidden_size_1 = hidden_size_1
    self.hidden_size_2 = hidden_size_2
    self.nb_layers_1 =nb_layers_1
    self.nb_layers_2 = nb_layers_2
    self.lstm_1 = nn.LSTM(self.nb_features, self.hidden_size_1, self.nb_layers_1, dropout=dropout)  #, dropout=dropout
    self.lstm_2 = nn.LSTM(self.hidden_size_1, self.hidden_size_2, self.nb_layers_2, dropout=dropout)
    self.lin = nn.Linear(self.hidden_size_2, 1)

def forward(self, input):
    h0 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
    h1 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
    #print(type(h0))
    c0 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
    c1 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
    #print(type(c0))

    output_0, hn_0 = self.lstm_1(input, (h0, c0))
    output, hn = self.lstm_2(output_0, (h1, c1))
    out = torch.tanh(self.lin(output[-1])) ##########out = self.lin(output[-1])
    #out = self.lin(output_2[-1])
    return out

The code can be run, however, the output is a straight line even tuning the hyperparameter (learning rate, dropout, activation method) and increases epoch (i.e. 3000 epochs), the output result was shown in the following.

f21072×774 130 KB

Could you please give me some suggestions to solve this problem. many thanks