Thank you for your answer. So is there a valid approach to be sure that those weights get freezed with using Adam?
Would it be possible to do it like Keras does it with its kernels?
To use parallel layers somehow, like one part for normal input and the other for conditions, so you can set requires_grad to False for the condition-part layer?