This should also work with detach. Along as you don’t call two times backward on the same gradient-history (because the history is deleted during backward()). If you want to backward the same history (or parts of the history) multiple times (as it is usually done in recurrent networks) you have to specify retain_graph=True in loss_value.backward()