Solved by ptrblck in post #7 
              
              
                Since the computation graph will be created dynamically during the forward pass, only the parameters will be updated (and get a valid gradient) which were used to calculate the loss. 
I.e. if you only use path1 during training, only self.path1 will be upgraded, while self.path2 keeps its initial val…