Solved by Oli in post #2 
              
              
                You can freeze the parameters by setting their requires_grad to False. Then you can set them to True once to want to update them again 
for param in A_branch.parameters():
    param.requires_grad = False