Solved by jpeg729 in post #2 
              
              
                The LSTM normally expects the batch dimension to come second, but I guess your dataloader is providing data with the batch dimension first. 
If I have correctly understood the problem this should correct it. 
self.lstm = nn.LSTM(n_fac,hidden_dim, batch_first=True)