hi,
i have a c++ loss-wrapped in python.
here is some stats:
in all these cases, ddp is used. but we can choose to use one or two gpus.
here we show the forward time in the loss. more specifically, part of the code in the forward.
that part operates on cpu. so, gpu is not involved since we convert the output gpu tensor from previous computation to cpu().numpy(). then, computations are carried on cpu.
time is measured using

def forward(x):
    torch.cuda.synchronize()
    start_event = torch.cuda.Event(enable_timing=True) 
    end_event = torch.cuda.Event(enable_timing=True)
    start_event.record()
    # cpu region ---
    compute_on_cpu()
    # cpu region ---
    end_event.record()
    torch.cuda.synchronize()
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print('time cpu: {}'.format(elapsed_time_ms))

1 gpu:

multi-threading is on: 70ms
multi-threading is off: 500ms.

2gpus:

multi-threading is on: 500ms
multi-threading is off: 500ms.

the loss uses the maximum number of threads (openmp omp_get_max_threads()) which is 48 in this case.
it looks like when using multi-processes (multi-gpus), multi-threading is not working…
any idea why (gil bottleneck???)? and how to fix this?
the whole point of using ddp is to speedup computations.
thanks