Solved by matthewleigh in post #2 
              
              
                Nevermind! I found the correct function to call: 
torch.set_grad_enabled(True)

and 
torch.set_grad_enabled(False)

do exactly what I want.