If shuffling interferes negatively with your training, it seems that your training routine or model depends on the sequential input of the data.
E.g. if you are using an RNN-like model, you might pass the batches sequentially and reusing the hidden states. In that case, shuffling the data might destroy the temporal correspondence.
You could try to shuffle “segments” of the data, but it depends of course on your actual use case.