So if it’s the same data and the same network, there must be some kind of discrepancy in your training vs test code.
I suggest the following check: first remove shuffle so you can load the train and “test” (which is just the train folder) in the same order. Print out torch.norm(data) for both cases, (actually verify they are the same input to the network). If they are, and the output is different, try to step through the layer weights to see which are different. Good luck!