Some users report that the unusual memory consumption appears at the forward stage. There’re so many related functions in your code. Could you please try whether this issue happens on some simpler networks like AlexNet? In that way, I guess the problem may be on Conv or BatchNorm.