I am trying to do the simplest LSTM possible to predict a sequence. It works by loading a sequence in the LSTM module and then taking each item of the output through a linear model. I am training it 1000 times on the same sequence (0.05,0.1…1), that should become (0.1,0.05,…,1.1)
It does not really work. After 1000 iterations the loss is reduced a bit and the output resembles the right one in some ways. But it works really badly for having being trained so much on a sigle, simple, sequence.
I’d really appreciate any help in understanding why it performs so bad.
Here is the code:
import torch
import numpy as np

lstm = torch.nn.LSTM(input_size=1, hidden_size=100, num_layers=1, batch_first=True).to(torch.double)
linear = torch.nn.Linear(100,1).to(torch.double)
criterion = torch.nn.MSELoss()
params = list(lstm.parameters()) + list(linear.parameters())
optimizer = torch.optim.Adam(params, 1e-3)

L = 21

x = torch.tensor(np.arange(L)/20, dtype=torch.double).unsqueeze(0).unsqueeze(2)
y = torch.tensor(np.arange(2,L+2)/20,  dtype=torch.double)

for j in range(1000):
    hidden = (torch.zeros((1,1,100), dtype=torch.double), torch.zeros((1,1,100), dtype=torch.double))
    x = x
    y = y
    output, hidden = lstm(x, hidden)
    loss = 0.0
    for i in range(L):
        o = linear(output[:,i,:].squeeze())
        loss += criterion(o, y.squeeze()[i])
    print(loss)
    loss.backward()
    optimizer.step()

out_seq = torch.empty((L))
for i in range(L):
    o = linear(output[:,i,:].squeeze())
    out_seq[i] = o
    
print("output (expected): ", y)
print("output:", out_seq)


Output:
loss falls and rises, falls and rises again, but the trend is decreasing
output (expected):  tensor([0.1000, 0.1500, 0.2000, 0.2500, 0.3000, 0.3500, 0.4000, 0.4500, 0.5000,
        0.5500, 0.6000, 0.6500, 0.7000, 0.7500, 0.8000, 0.8500, 0.9000, 0.9500,
        1.0000, 1.0500, 1.1000], dtype=torch.float64)
output: tensor([ 0.1231, -0.2927, -0.2559, -0.0443,  0.0863,  0.1924,  0.2867,  0.3715,
         0.4473,  0.5152,  0.5764,  0.6319,  0.6827,  0.7296,  0.7734,  0.8146,
         0.8540,  0.8920,  0.9289,  0.9650,  1.0007], grad_fn=<CopySlices>)