Solved by alexis-jacq in post #4 
              
              
                When you did x = F.relu(self.lstm(inputs)) in your forward pass, you forgot to provide the lstm with hidden and cell units.