Posting a couple links that might help figure this out:
Example custom dataloader with pin_memory on individual examples or batches:

  
      github.com
  
  
    Cadene/vqa.pytorch/blob/master/vqa/lib/dataloader.py 13
import torch
import torch.multiprocessing as multiprocessing
from .sampler import SequentialSampler, RandomSampler
import collections
import math
import sys
import traceback
import threading
if sys.version_info[0] == 2:
    import Queue as queue
else:
    import queue


class ExceptionWrapper(object):
    "Wraps an exception plus traceback to communicate across threads"

    def __init__(self, exc_info):
        self.exc_type = exc_info[0]
        self.exc_msg = "".join(traceback.format_exception(*exc_info))


  This file has been truncated. show original

  
  
    
    
  
  


smth mentioning that DataParallel tries to use async:

Yes, DataParallel will try to use  async=True  by default.
DataParallel model and pin_memory() 12