Hi, I am trying to get a transformer to do some simple timeseries forecasting, but I am struggling with finding the right way to present the data to the network.
The input and target should have dimensions {batch, sequence, features}.
If I want to train it on a sequence (1,2,3,…) with the goal to predict the next number, how should I format the data?
My guess would be
[1,   2,  3, 4;
2,   3,  4, 5;
3,  4,  5, 6]
So I have feature size 3, and sequence size 4 in this case. The target would be shifted right one column:
[ 2,  3, 4, 5;
3,  4, 5, 6;
4,  5, 6, 7]
Then, to train the transformer something like:
tf_model = torch.nn.transformer()
opt = torch.optim.Adam(tf_model.parameters())
loss_fn = torch.CrossEntropyLoss() # Or other

out = tf_model(input,target);
loss=loss_fn(out,target)
loss.backward()
optimizer.step()

Can anyone point me to some examples where the transformer module is used on something else than NLP?