@odats can you set a seed each epoch for that ?
@trainer.on(Events.EPOCH_STARTED)
def set_epoch_seed():
    set_seed(trainer.state.epoch)

If this does not work for you, please provide a minimal code snippet to see the problem.