Hi,everyone
I run my code on a node with 8 GPUs. I have several observations below.

num_worker = 32 for both training loader and testing loader. In first epoch, it’s fine during training,  but the error occurs during testing after the epoch.
num_worker = 64 for both training loader and testing loader. Got the same results as (1).
64 and 8 for training loader and testing loader respectively. First epoch is fine, but the error occurs in the training of next epoch.
32 and 8 for training loader and testing loader respectively. Got the same results as (3).
num_worker = 16 for both training loader and testing loader. Everything is fine.
cat /proc/sys/kernel/pid_max: 32768;
memory usage is shown below with  num_worker = 32 for both training loader and testing loader.

image.png996×113 5.72 KB



This is my traceback:
Traceback (most recent call last):
  File "train.py", line 225, in <module>
    trainer.train()
  File "train.py", line 164, in train
    acc = self.evaluate(epoch)
  File "train.py", line 198, in evaluate
    for idx, (img, label) in enumerate(self.loader_test):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 560, in __init__
    w.start()
  File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/opt/conda/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/conda/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

What’s the possible causes?