Thank you for your answer. I actually found my own solution to produce the vector of indexes of the words like you input tensor. I processed 8 different books, I created a flat corpus of all the books and then I decided to create fixed sentences of 8 words each. I have done a shuffe of these sentences in order to create training validation and test. I built a method which take the sentence and converts it into a tensor of indexes witch is passed to the LSTM but I’m missing a crucial point. Suppose that the sentence is “The dog is on the table” which converts to [10,0,34,983,8,12]. Then i pass this vector to the LSTM. My objective is to predict for the word ‘The’ the word ‘dog’ for the word ‘dog’ the word ‘is’ and so on. Is there an efficient way to this? I have to create all the couples? what’s the output of the LSTM? I’m missing these concepts.
Also I noticed that the LSTM gives me in output a tensor with a length equal to the length of input tensor while I’m expeting in output a tensor of length = len(input) - 1 since the last word does not have a ground truth.
Also, how long need to be each tensor in output? Actually each output tensor has a length equal to the vocabulary because I thought that I use softmax fo selecting the index with the highest probability. Is that correct?
Final question, I obtained tensors of different length because some words are not present in my vocabulary, which is the best solution for having tensors of the same length?
UPDATE
I found a temporary solution for the previous problems. Now I’m feeding the LSTM with sequences of vectors with fixed length = 7. Using a batch_size = 256 I have in input to the model a tensor of size [256,7]. For simplicity I’m using for the moment a batch_size of 1 so I have [1,7]. Each of my embedding has a length of 200, but when i give in input for example a tensor like [1,0,1,89,177,7,7] which are the indexes of the words in the embeddings matrix it gives me the following error:
Expected target size (1, 200), got torch.Size([1, 7])
I set input_dim = 200 and layers_num = 7 because i need to predict seven words sequentially, but I think that I miss understood the meaning of these parameters
My current LSTM
class lstm(nn.Module):

    def __init__(self, input_size, hidden_units, layers_num, matrix_embeddings, dropout_prob=0):
        
        super().__init__()
        #embedding
        self.embeddings = nn.Embedding.from_pretrained(matrix_embeddings)
        self.embeddings.requires_grad = False

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.rnn = nn.LSTM(input_size=input_size, 
                           hidden_size=hidden_units,
                           num_layers=layers_num,
                           dropout=dropout_prob,
                           batch_first=True)

        # The linear layer that maps from hidden state space to tag space
        self.out = nn.Linear(hidden_units, input_size)

    def forward(self, sentence, state=None):
        embeds = self.embeddings(sentence)
        # LSTM
        x, rnn_state = self.rnn(embeds, state)
        # Linear layer
        x = self.out(x)
        return x, rnn_state