You could try to use the output of your “program” e.g. as a target for an autoencoder-like architecture, where the label might be fed to a side branch coming from the latent tensor, while the generator output is trained with e.g. nn.MSELoss.
I assume your program cannot generate valid gradients, which could be fed to the generator?