1 Trying to load data onto memory
  2 Preparing data from file = trg_data.txt
  3 CUDA error: out of memory
  4 Not enough memory to load all the data to GPU. Run script without the '-m' flag
  5 torch.cuda.max_memory_allocated()=0 ,torch.cuda.max_memory_cached() = 0
  6 
  7 Preparing data from file = trg_data.txt
  8 THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMath.cu line=35 error=2 : out of memory
  9 Traceback (most recent call last):
 10   File "train_ext.py", line 239, in <module>
 11     intents_to_train=args.intents, debug_mode=args.debug, test=args.test, load_to_memory=args.load_all_data_to_memory    )   
 12   File "train_ext.py", line 59, in train
 13     num_samples_key='num_words', get_raw_samples_while_iter=debug_mode, debug_mode=debug_mode)
 14   File "/media/storage/dev/utils.py", line 261, in __init__
 15     self.features_vec = torch.zeros((self._batch_size, self.max_seq_len), dtype=torch.long, device=self.device)
 16 RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCTensorMath.cu:35

here’s the error. Line 4 is where OOM occurs for DataLoader1 and from Line 7 DataLoader2 tries to take over.