Welp, just trying to add to the discussion here. Found this in the docs 40
In-place operations can potentially overwrite values required to compute gradients.

I still would love to know if it offers a speed up to use inplace for inference, or in layers without weights. Or could it lower the memory footprint?