Now, I have another general question. I am setting shuffle=True for my training dataset loader. But anyhow what comes out of the training loader will be of the 4D size [batch_size, num_steps, data_0, data_1]. I need to shuffle this tensor along the 2nd dimension as mentioned before. Then I will unsqueeze and add extra dimension so the tensor will be [batch_size, num_steps, 1, data_0, data_1]. Now I will send this tensor to my model which is a CNN and I will iterate over the num_steps step by step in the forward path such that the input to the CNN is just 1 channel. I stack the output for each step in a way that the output of the forward path will be ``[num_steps, batch_size, number_of_classes]. For the loss calculation I will then do as well loss for each num_step some up thier losses then do backpropagation.  So. from my point of view as I am calculating the loss for each step the final loss should also be the same whether I shuffled or not, and the parameters should be updated in the same way.
My aim is then to find that the CNN without shuffeling and with shuffeling should give me the same final accuracies as it should not be remebering any kind of memory. That is why I need to shuffle and check. But I made one approach now and I found that when I shuffled a long the num_steps the final accuracy changed as well? does that make any sense to you?