So after changing stuff a bit (adding a SLOGAN single-sided Lipschitz objective penalty, BatchNorm->InstanceNorm) and so, it would seem that the normalization matters a lot less. This is with bad=True after ~ 550 iters.
Best regards
Thomas