text: Title: [Bug] xtts OrderedVocab problem
  Body: ### Describe the bug
  
   > TRAINING (2023-10-28 18:37:37)
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  
  ### To Reproduce
  
  training xtts with standard recipe
  
  ### Expected behavior
  
  _No response_
  
  ### Logs
  
  ```shell
  python finetunextts.py
  >> DVAE weights restored from: C:\Users\someone\Desktop\xtts/run\training\XTTS_v1.1_original_model_files/dvae.pth
   | > Found 489 files in C:\Users\someone\Desktop\xtts
  fatal: not a git repository (or any of the parent directories): .git
  fatal: not a git repository (or any of the parent directories): .git
   > Training Environment:
   | > Current device: 0
   | > Num. of GPUs: 1
   | > Num. of CPUs: 12
   | > Num. of Torch Threads: 1
   | > Torch seed: 1
   | > Torch CUDNN: True
   | > Torch CUDNN deterministic: False
   | > Torch CUDNN benchmark: False
   > Start Tensorboard: tensorboard --logdir=C:\Users\someone\Desktop\xtts/run\training\GPT_XTTS_LJSpeech_FT-October-27-2023_10+56PM-0000000
  
   > Model has 543985103 parameters
  
   > EPOCH: 0/1000
   --> C:\Users\someone\Desktop\xtts/run\training\GPT_XTTS_LJSpeech_FT-October-27-2023_10+56PM-0000000
   > Filtering invalid eval samples!!
   > Total eval samples after filtering: 4
  
   > EVALUATION
  
   | > Synthesizing test sentences.
  
    --> EVAL PERFORMANCE
       | > avg_loader_time: 0.01900 (+0.00000)
       | > avg_loss_text_ce: 0.04067 (+0.00000)
       | > avg_loss_mel_ce: 4.33739 (+0.00000)
       | > avg_loss: 4.37806 (+0.00000)
  
   > BEST MODEL : C:\Users\someone\Desktop\xtts/run\training\GPT_XTTS_LJSpeech_FT-October-27-2023_10+56PM-0000000\best_model_0.pth
  
   > EPOCH: 1/1000
   --> C:\Users\someone\Desktop\xtts/run\training\GPT_XTTS_LJSpeech_FT-October-27-2023_10+56PM-0000000
   > Sampling by language: dict_keys(['en'])
  
   > TRAINING (2023-10-27 22:57:08)
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5024, your vocabulary could be corrupted !
  The OrderedVocab you are attempting to save contains a hole for index 5025, your vocabulary could be corrupted !
  
     --> STEP: 0/243 -- GLOBAL_STEP: 0
       | > loss_text_ce: 0.04536  (0.04536)
       | > loss_mel_ce: 4.79820  (4.79820)
       | > loss: 4.84356  (4.84356)
       | > current_lr: 0.00001
       | > step_time: 0.88430  (0.88431)
       | > loader_time: 70.14840  (70.14841)
  
  
     --> STEP: 50/243 -- GLOBAL_STEP: 50
       | > loss_text_ce: 0.04994  (0.04525)
       | > loss_mel_ce: 5.39171  (4.74854)
       | > loss: 5.44165  (4.79379)
       | > current_lr: 0.00001
       | > step_time: 0.66870  (1.54556)
       | > loader_time: 0.01600  (0.01624)
  
  
     --> STEP: 100/243 -- GLOBAL_STEP: 100
       | > loss_text_ce: 0.04045  (0.04345)
       | > loss_mel_ce: 3.84910  (4.67366)
       | > loss: 3.88955  (4.71711)
       | > current_lr: 0.00001
       | > step_time: 1.74700  (1.66512)
       | > loader_time: 0.01520  (0.01434)
  
  
     --> STEP: 150/243 -- GLOBAL_STEP: 150
       | > loss_text_ce: 0.05477  (0.04379)
       | > loss_mel_ce: 5.39814  (4.72587)
       | > loss: 5.45292  (4.76966)
       | > current_lr: 0.00001
       | > step_time: 2.80970  (1.85835)
       | > loader_time: 0.01400  (0.01352)
  
  
     --> STEP: 200/243 -- GLOBAL_STEP: 200
       | > loss_text_ce: 0.03867  (0.04367)
       | > loss_mel_ce: 4.21473  (4.71702)
       | > loss: 4.25340  (4.76068)
       | > current_lr: 0.00001
       | > step_time: 3.30200  (2.20536)
       | > loader_time: 0.00500  (0.01207)
  
   > Filtering invalid eval samples!!
   > Total eval samples after filtering: 4
  
   > EVALUATION
  
   | > Synthesizing test sentences.
  
    --> EVAL PERFORMANCE
       | > avg_loader_time: 0.01202 (-0.00698)
       | > avg_loss_text_ce: 0.03961 (-0.00106)
       | > avg_loss_mel_ce: 4.15599 (-0.18140)
       | > avg_loss: 4.19560 (-0.18246)
  
   > BEST MODEL : C:\Users\someone\Desktop\xtts/run\training\GPT_XTTS_LJSpeech_FT-October-27-2023_10+56PM-0000000\best_model_243.pth
  ```
  
  
  ### Environment
  
  ```shell
  {
      "CUDA": {
          "GPU": [
              "NVIDIA GeForce RTX 3060"
          ],
          "available": true,
          "version": "11.7"
      },
      "Packages": {
          "PyTorch_debug": false,
          "PyTorch_version": "2.0.1",
          "TTS": "0.19.0",
          "numpy": "1.22.0"
      },
      "System": {
          "OS": "Windows",
          "architecture": [
              "64bit",
              "WindowsPE"
          ],
          "processor": "AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD",
          "python": "3.9.13",
          "version": "10.0.22621"
      }
  }
  ```
  
  
  ### Additional context
  
  _No response_