Ok, my theory has been dismantled. I tried the following using brand new pytorch 0.3 functionality:

seg_out = <Network outputs one-hot>
mask = Variable(<A batch of binary masks BxHxW>, requires_grad=False)
seg_gt = Variable(<Ground truth labels>,  requires_grad=False)
seg_loss  = ce_loss_criterion(seg_out, seg_gt, reduce=False)
seg_loss[mask == 1] = seg_loss[mask == 1] * weight_2
seg_loss[mask == 0] = seg_loss[mask == 0] * weight_1
seg_loss = seg_loss.mean()
seg_loss.backward()
opt.step()

But it still randomly runs out of memory. Very disturbing to see that if you do something that is a little bit different from the standard vanilla pipeline (network -> prediction -> crossentropy) you get troubles after troubles. This normally indicates that test coverage is very thin.