However, I’ve just seen that your concatenating the MNIST and CIFAR results at dimension 1 from the classifier. I’m not sure if that’s intentional.

Yes, it is intentional. I want a multi input network. I am doing an experiment of meta-transfer learning, in which I sum weights of 2 networks basically. And now i am trying to have a multi input network trained on 2 datasets.
You are right, I am defining a new loss function for this experiment, I will not use BCEWithLogitLoss
So, do you suggest to modify something in the net, or I can work on my custom loss function?
My CustomLoss will be this: CrossEntropy(MNIST) + CrossEntropy(CIFAR10) + CrossEntropy([MNIST+CIFAR10]-MNIST) (this last term of loss should guarantee that the network on MNIST have similar performances before and after the operation of sum.
Tomorrow I will try to implement something like this, and probably I will ask for help here 
Edit:
@Unity05 do you think a solution like this could be good?
class CustomLossFunction(nn.Module):
    def __init__(self):
        super(CustomLossFunction, self).__init__()
    def forward(self, pred_mnist, pred_cifar, target_mnist, target_cifar):
        loss_mnist = nn.BCEWithLogitsLoss(pred_mnist, target_mnist)
        loss_cifar = nn.BCEWithLogitsLoss(pred_cifar, target_cifar)
        loss_mnist_cifar = nn.BCEWithLogitsLoss(pred_mnist_cifar - pred_mnist, target_mnist_cifar - target_mnist)
        loss_sum = loss_mnist + loss_cifar + loss_mnist_cifar
        return loss_sum