I figured out why this happens. There is a line 1783 in transformers/models/unispeech_sat/modeling_unispeech_sat.py:
output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states
So even if you pass output_hidden_states=None in forward pass, the output will anyways contain hidden states (because config.use_weighted_layer_sum=True by default). And if you are using output in a form such like that:
@dataclass
class SpeechClassifierOutput(ModelOutput):
    loss: Optional[torch.FloatTensor] = None
    logits: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
    attentions: Optional[Tuple[torch.FloatTensor]] = None

This will actually contain all the hidden states.
Evaluation round will accumulate all the hidden states for all samples, therefore you will run out of memory. So I just manually set hidden_states=None in the output.
Hopefully this will help someone.