I would like to train a “generator” network that receives an RGB image (e.g. from ImageNet) and generates an output image with these criteria:

The output is as similar as possible to the original image.
The accuracy of one network (e.g. resnet50) is dropped by this “output”.
The accuracy of another network (e.g. inception_v3) remains intact by this “output”.

The snipped of code would be like this:
model_gen.zero_grad()
output_imgs = model_gen(input_imgs)
loss_gen = MSELoss(output_imgs, input_imgs)

output_n1 = resnet50(output_imgs)
loss_n1 = CrossEntropyLoss(output_n1, targets)

output_n2 = inception_v3(output_imgs)
loss_n2 = CrossEntropyLoss(output_n2, targets)

loss_all = loss_gen + (1 / loss_n1) + loss_n2
loss_all.backward()
optimizer.step()

With this procedure, my generator after 1 epoch lerans to apply only subtle changes to pixle values, therferore “loss_gen” becomes almost 0, however it deosn’t learn to increase “loss_n1” (since it’s in the dominator) and decrease “loss_n2”.
Essentially, my networks only learn to replicate the output images from input images, and accuracy of resnet50 and inception_v3 with respect to images of my generator remains in par.
I appreciate any suggestions along this line