Hi,
I have a i7 quad-core cpu running at 4.2GHz, with 3 x NVIDIA TITAN V GPUs. While running my training loop for a U-Net model, with the following batch sizes, a performance profile shows that the code spends 62.8% of its time in method acquire of _thread.lock
  # training parameters
  train:
    batch_size: 20
    num_workers: 4

  # validation parameters
  valid:
    batch_size: 6
    num_workers: 4


201806081050-train-unetv2-pp-acquire-thread-lock-62.png1589×237 25.6 KB

Is this somehow PyTorch related, with respect to the DataLoader automatically spinning up worker threads and trying to allocate threads for processing?
Each of my GPUs have 12GB of memory, but they are effectively operating at 1% utilization.
If I reduce the batch size to 1 with 4 workers each, the acquire thread lock still take up most of the time at 50%:
  # training parameters
  train:
    batch_size: 1
    num_workers: 4

  # validation parameters
  valid:
    batch_size: 1
    num_workers: 4


201806081100-train-unetv2-pp-acquire-thread-lock-58-b1-w4.png905×340 31.7 KB

It gets weirder when I reduce both batch size and num of workers to 1, the acquire thread.lock method ends up consuming 71.1% of the total time.

201806081110-train-unetv2-pp-acquire-thread-lock-58-b1-w1.png911×339 32.1 KB

I got better results I matched the number of threads with the number of samples in the batch with the number of GPUs available. The acquire _thread.lock method was at 47.1% with 5.08secs/iteration.
  # training parameters
  train:
    batch_size: 3
    num_workers: 3

  # validation parameters
  valid:
    batch_size: 2
    num_workers: 2


201806081128-train-unetv2-pp-acquire-thread-lock-58-b3-w3.png911×500 46.9 KB