Hello,
I am aware that there are several other topics about slow computation on multi GPU - none of them seems to fit to the behaviour I am facing.
I have a model that utilizes Huggingface BERT and has a total of 111,450,730 params. When I set  BERTs requires_grad to False, the model has 2,369,386 params. Apart from the number of parameters, the problematic behaviour is the same: BCE Loss computation is up to three times as expensive when switching to Dataparallel using two instead of one GPU (also changing the batch size does not have any effect).
My model is pretty much standard, however F.binary_cross_entropy(pred, target) takes ~50 ms in single GPU mode and ~150 ms when training on two GPUs.
def update_model_for_trainer(batch):
    model.train()
    optimizer.zero_grad()
    input, target = batch
    pred = model(input)
    loss = F.binary_cross_entropy(pred, target) # 50 ms without and 150 ms with Dataparallel
    loss.backward()
    optimizer.step()
    return loss.item()

Things tried so far:

Explicitely moving prediction and target to cuda:0

Integrating the loss calculation into the model, after that I used sum(loss).backwards(); had no effect

I would be really thankful for some tips what to do next to investigate the problem.
Thank you in advance
Thomas