First of all, there are no activation layers in the given code for conv2 sequential block (which is strange).

Ho I was looking at the net in the first post. But indeed, if there is no activation, this is a moot argument.

it is their job to save the input.

Actually it is better to save the output when they can (which relu can do) because in things like
conv1 -> relu -> conv2 -> relu, etc. then the output of the conv don’t need to be saved, only the input! So you actually only save half the number of intermediary results compared to the case where the relu would save its input.
And yes inplace relu makes this point less important as the two intermediary results are just one and so just one is saved.

Surely you do need inputs to compute weights gradients. But in this case, weights have flag requires_grad=False so these gradients will not be computed.

One issue that arise here is that if you use optimized libraries like cudnn, they provide a single backward function for both input and weight. And we have no control over it in PyTorch. So, in some cases, even though the weights don’t require gradients, we need to save extra informations to be able to make these third_party library functions run properly.
Also in some cases, we could improve our codegen to be smarter on what needs to be saved, you can follow up the corresponding issue here: https://github.com/pytorch/pytorch/issues/42371 4