Your test/validation data is meant to measure how good your model is. Hence, choosing to shuffle or no not your test/validation data set depends on what you are choosing to measure.
For example, if you are extracting a information that is batch dependent (e.g mean loss on the batch) or you have a component in your neural network that is batch dependent, then you should shuffle for consistency.