Yes it should.
If you re-enable the shuffling the error happens again? Can you add some printing to check if all the Tensors have the right value.
If you can make a small code sample (~30 lines) that reproduce the problem, that would be very helpful !