text: Title: Training crashes during pretraining
  Body: 
  After extracting the version 01_04_2021, I immediately ran the pretraining with default options.
  
  I set the target iterations to 500,000.
  
  After loading and training for a while it crashes, I restarted it each time reducing batch size and it still crashes.
  
  The final output:
  
      Initializing models: 100%|#############################################| 5/5 [00:03<00:00,  1.64it/s]
      Loaded 15843 packed faces from D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\pretrain_CelebA
      Sort by yaw: 100%|################################################| 128/128 [00:00<00:00, 295.61it/s]
      Sort by yaw: 100%|################################################| 128/128 [00:00<00:00, 289.58it/s]
  
      =============== Model Summary ===============
      ==                                         ==
      ==            Model name: basic_SAEHD      ==
      ==                                         ==
      ==     Current iteration: 91392            ==
      ==                                         ==
      ==------------- Model Options -------------==
      ==                                         ==
      ==            resolution: 128              ==
      ==             face_type: wf               ==
      ==     models_opt_on_gpu: True             ==
      ==                 archi: liae-ud          ==
      ==               ae_dims: 256              ==
      ==                e_dims: 64               ==
      ==                d_dims: 64               ==
      ==           d_mask_dims: 22               ==
      ==       masked_training: True             ==
      ==       eyes_mouth_prio: False            ==
      ==           uniform_yaw: True             ==
      ==             adabelief: True             ==
      ==            lr_dropout: n                ==
      ==           random_warp: False            ==
      ==       true_face_power: 0.0              ==
      ==      face_style_power: 0.0              ==
      ==        bg_style_power: 0.0              ==
      ==               ct_mode: rct              ==
      ==              clipgrad: True             ==
      ==              pretrain: True             ==
      ==       autobackup_hour: 1                ==
      == write_preview_history: False            ==
      ==           target_iter: 500000           ==
      ==           random_flip: True             ==
      ==            batch_size: 4                ==
      ==             gan_power: 0.0              ==
      ==        gan_patch_size: 16               ==
      ==              gan_dims: 16               ==
      ==                                         ==
      ==-------------- Running On ---------------==
      ==                                         ==
      ==          Device index: 0                ==
      ==                  Name: GeForce RTX 2060 ==
      ==                  VRAM: 6.00GB           ==
      ==                                         ==
      =============================================
      Starting. Target iteration: 500000. Press "Enter" to stop training and save model.
      [18:32:21][#094491][0288ms][0.4154][0.3854]
      [18:47:21][#097587][0281ms][0.4149][0.3843]
      [19:02:21][#100685][0300ms][0.4120][0.3807]
      [19:17:22][#103782][0305ms][0.4106][0.3801]
      Error: 46][#104277][0275ms][0.4674][0.3715]
      Traceback (most recent call last):
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\mainscripts\Trainer.py", line 130, in trainerThread
          iter, iter_time = model.train_one_iter()
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\models\ModelBase.py", line 462, in train_one_iter
          losses = self.onTrainOneIter()
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 676, in onTrainOneIter
          (warped_dst, target_dst, target_dstm, target_dstm_em) ) = self.generate_next_samples()
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\models\ModelBase.py", line 449, in generate_next_samples
          sample.append ( generator.generate_next() )
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\samplelib\SampleGeneratorBase.py", line 21, in generate_next
          self.last_generation = next(self)
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 112, in __next__
          return next(generator)
        File "D:\DeepFaceLab_NVIDIA_01_04_2021\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 73, in __next__
          gen_data = self.cs_queue.get()
        File "multiprocessing\queues.py", line 94, in get
        File "multiprocessing\connection.py", line 216, in recv_bytes
        File "multiprocessing\connection.py", line 318, in _recv_bytes
        File "multiprocessing\connection.py", line 340, in _get_more_data
      MemoryError
     
  
  System Information:
  
      Edition	Windows 10 Home
      Version	20H2
      Installed on	‎7/‎21/‎2020
      OS build	19042.746
      Experience	Windows Feature Experience Pack 120.2212.551.0
      Processor	AMD Ryzen 7 4800H with Radeon Graphics            2.90 GHz
      Installed RAM	16.0 GB (15.4 GB usable)
  
  nvcc output:
  
      nvcc: NVIDIA (R) Cuda compiler driver
      Copyright (c) 2005-2020 NVIDIA Corporation
      Built on Mon_Oct_12_20:54:10_Pacific_Daylight_Time_2020
      Cuda compilation tools, release 11.1, V11.1.105
      Build cuda_11.1.relgpu_drvr455TC455_06.29190527_0
  
  nvidia-smi output:
  
      Thu Jan 14 19:53:32 2021
      +-----------------------------------------------------------------------------+
      | NVIDIA-SMI 460.89       Driver Version: 460.89       CUDA Version: 11.2     |
      |-------------------------------+----------------------+----------------------+
      | GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
      | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
      |                               |                      |               MIG M. |
      |===============================+======================+======================|
      |   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0 Off |                  N/A |
      | N/A   44C    P8     9W /  N/A |    164MiB /  6144MiB |      0%      Default |
      |                               |                      |                  N/A |
      +-------------------------------+----------------------+----------------------+
      
      +-----------------------------------------------------------------------------+
      | Processes:                                                                  |
      |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
      |        ID   ID                                                   Usage      |
      |=============================================================================|
      |  No running processes found                                                 |
      +-----------------------------------------------------------------------------+
  
  