Hm okay interesting, so it should work…
I also thought about another idea: To have two parallel layers for the input, one for the normal input data and one for the condition input. So at the end of both layer calculations I could just add the two outputs together and use that for the next layers as input. In this way I could set the requires_grad flag to False for the first parallel layer that is responsible for the normal input data and leave the requires_grad flag for the second parallel layer that is responsible for the conditions at True.
Does that make sense to you? Or do you see any problems there?
Thanks for your help again and sorry that Im asking so much, but Im pretty new to pytorch