I’m looking at implementing a simple character RNN to learn about RNNs and PyTorch at the same time.  I’m looking at the example here:

  
      github.com
  
  
    vinhkhuc/PyTorch-Mini-Tutorials/blob/master/6_lstm.py 20
from __future__ import division
import numpy as np

import torch
from torch.autograd import Variable
from torch import optim, nn

from data_util import load_mnist


class LSTMNet(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(LSTMNet, self).__init__()
        self.hidden_dim = hidden_dim
        self.lstm = nn.LSTM(input_dim, hidden_dim)
        self.linear = nn.Linear(hidden_dim, output_dim, bias=False)

    def forward(self, x):
        batch_size = x.size()[1]
        h0 = Variable(torch.zeros([1, batch_size, self.hidden_dim]), requires_grad=False)


  This file has been truncated. show original

  
  
    
    
  
  


If a sequence vector and its y values from training data look like: x =>  ‘a b c d e’, y => ‘b c d e f’
The forward() method on LSTM returns a tensor of shape (sequence_length, num sequences in batch, input size).   If the vocabulary size is 26, and we are using a batch of one, then the lstm’s forward() will return a tensor of shape (5, 1, 26).
My question is, do we compute the loss of this tensor vs all five elements in y?  or just the last element?  The examples above look like the return just the last predicted element from forward():

  
      github.com
  
  
    vinhkhuc/PyTorch-Mini-Tutorials/blob/master/6_lstm.py#L23 2

    super(LSTMNet, self).__init__()
    self.hidden_dim = hidden_dim
    self.lstm = nn.LSTM(input_dim, hidden_dim)
    self.linear = nn.Linear(hidden_dim, output_dim, bias=False)


def forward(self, x):
    batch_size = x.size()[1]
    h0 = Variable(torch.zeros([1, batch_size, self.hidden_dim]), requires_grad=False)
    c0 = Variable(torch.zeros([1, batch_size, self.hidden_dim]), requires_grad=False)
    fx, _ = self.lstm.forward(x, (h0, c0))
    return self.linear.forward(fx[-1])




def train(model, loss, optimizer, x_val, y_val):
x = Variable(x_val, requires_grad=False)
y = Variable(y_val, requires_grad=False)


# Reset gradient
optimizer.zero_grad()


# Forward



  
  
    
    
  
  


Then this must get loss computed vs the last element in y, ‘f’.  (If I understand this right!)  Is this what we want?  Don’t we want to reinforce the predictions made by each item in the sequence?
Thanks for your help and apologies if I’m way  off.