Yes, you need gradiens to backprop. I don’t know which kind of predictions your netowrk is doing but it sounds like you need to pose your problem in function of the network’s predictions rather than blender.
Anyway controlling a software sound like a very complicated task far away of what you can do just with a CNN. (You can google reinforcement learning)
Anyway, it seems you are getting a “command” from the cnn which later on you pass to blender. Then you try to match the result and a target “image”. You should try to pose the problem in a different way, where you know which command the network should predict to generate the desired effect and use that as target.