Thanks for the clarification! I think this behavior suggests that even with no_grad, extra memory is allocated first and the deallocated in the caching allocator rather than being skipped. I wonder if this is true for other layers as well…