text: Title: High Memory consumption on ASR inferencing
  Body: ## ❓ Questions and Help
  
  #### What is your question?
  High memory consumption on ASR inferencing on a 16GB RAM server. It used up the system RAM up to 15.67GB and cause the whole server to hang.
  
  ![image (1)](https://github.com/facebookresearch/fairseq/assets/51883244/b0e990a0-a5c8-4430-834c-d98955a0b1ce)
  
  
  #### What's your environment?
  
   - fairseq Version (e.g., 1.0 or main): main
   - PyTorch Version (e.g., 1.0)
   - OS (e.g., Linux): Linux
   - How you installed fairseq (`pip`, source): source
   - Build command you used (if compiling from source):
   ```
  cd /path/to/fairseq-py/
  python examples/mms/asr/infer/mms_infer.py --model "/path/to/asr/model" --lang lang_code \
    --audio "/path/to/audio_1.wav"
  ```
   - Python version: 3.8
   - CUDA/cuDNN version: None
   - GPU models and configuration: None
   - Any other relevant information: I am using docker.
  