I tried the following:




items
num_workers=1
num_workers = 2
num_workers = 4
num_workers = 8




CPU
10700K
10700K
10700K
10700K


CPU   RAM(G)
16
16
48
48


CPU RAM   used(G)
9.5
9.5
12
CUDA out of memory issue


GPU
3090
3090
3090
3090


GPU   RAM(G)
24
24
24
24


GPU RAM   used(G)
17.5
17.5
17.9
CUDA out of memory issue


GPU   Power(w)
257
357
357
CUDA out of memory issue


image count
10180
10180
10180
10180


epoch
100
100
100
100


batch   size
32
32
32
32


batch   count
319
319
319
319


total   time for training(hours)
15
6.459
6.05



time for   each epoch(minutes)
9
3.875
3.6333



average   time for batch(second)
1.69
0.73
0.617




I have a few questions:


the training speed of num_workers=2 is quite faster than that of num_workers=1, however, there is small difference for num_workers=2 and 4.
when num_workers=4, there are much CPU ram and GPU ram left, num_workers=8 doesn’t work. it is very weird.
num_workers assigns a few threads to load image to RAM(CPU). If batch size keeps 32, the RAM needed for CPU should not change. however, more RAM is used for num_workers = 4.


The following is the issue for num_workers=8:
Analyzing anchors... anchors/target = 5.86, Best Possible Recall (BPR) = 1.0000
Image sizes 640 train, 640 test
Using 4 dataloader workers
Logging results to runs\train\exp7
Starting training for 100 epochs...
     Epoch   gpu_mem       box       obj       cls     total   targets  img_size
  0%|          | 0/319 [00:00<?, ?it/s]Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
NumExpr defaulting to 8 threads.
  0%|          | 0/319 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "D:/code_python/har_hailiang/har_hdd/algo/train.py", line 288, in train
    pred = model(imgs)  # forward
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\code_python\har_hailiang\har_hdd\algo\models\yolo.py", line 122, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "D:\code_python\har_hailiang\har_hdd\algo\models\yolo.py", line 138, in forward_once
    x = m(x)  # run
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\code_python\har_hailiang\har_hdd\algo\models\common.py", line 35, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 3.85 GiB (GPU 0; 24.00 GiB total capacity; 1.47 GiB already allocated; 20.35 GiB free; 1.54 GiB reserved in total by PyTorch)
python-BaseException