KFrank:

Should the update you apply to your
prediction-network parameters depend in any way on the output
of the simulation layer?


The flow of my network is in a way where parameters are learnt in neural network. In a simple way, it can be compared with normal GAN where generator here does not generate image but just bunch of parameters. These parameters are updated by calculating losses by comparing simulated image with true image in discriminator.



 KFrank:

What role
does your simulation play in telling you whether your prediction
network is performing well or poorly?


Simulator generates images from learnt parameters from my prediction network (Generator), and this simulated image from my simulation algorithm is then compared with true image in discriminator. Losses are also calculated here. So, in a way, network is not important at all in simulation stage. I hope this answers it. But when I tried developing network like in my sample code, it is not updating any weights for my prediction parameters part. (Generator class weights)
To summarize,
gen = Generator()
learnt_params = gen(noisy_input)
#Generate simulated image
fake= simulate_image(learnt_params)
disc_real = disc(real).view(-1)
lossD_real = criterion(disc_real, torch.ones_like(disc_real))
disc_fake = disc(fake).view(-1)

 lossD_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
 lossD = (lossD_real + lossD_fake) / 2
 disc.zero_grad()
 lossD.backward(retain_graph=True)
 opt_disc.step()
### Train Generator: min log(1 - D(G(z))) <-> max log(D(G(z))
output = disc(fake).view(-1)
lossG = criterion(output, torch.ones_like(output))


I hope the issue is more clear now.
Thank yo



 KFrank:

Should the update you apply to your
prediction-network parameters depend in any way on the output
of the simulation layer?


Yes, simulated image generated using simulation layer (only numpy functions) will be compared with real image in discriminator stage to train it and then use discriminator to calculate losses for generator.
In simple words, I am using actual GAN structure, but instead of generator generating whole image from noise, I use generator to generate parameters which I use in simulation layer.
I hope this is more clear now.



 KFrank:

What role
does your simulation play in telling you whether your prediction
network is performing well or poorly?


Role of simulation is just to simulate image from predicted parameters, but it helps calculating losses as I explained before. I dont have any real known values for parameters and I can only compare real images and simulated images for their distribution matching.
Here is the sample code…
fake_params = gen(params)
simulated_image = simulate_image(fake_params) #consists of only numpy functions
disc_real = disc(real)
disc_fake = disc(fake)
lossD_real = criterion(disc_real, torch.ones_like(disc_real))
lossD_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
lossD = (lossD_real + lossD_fake) / 2
disc.zero_grad()
lossD.backward(retain_graph=True)
opt_disc.step()
output = disc(fake)
lossG = criterion(output, torch.ones_like(output))
gen.zero_grad()
lossG.backward()
opt_gen.step()

I hope now the connection and role of the simulator are more clear.