But doing out = in[:, :, 1:, 1:], will not give you in-place operation and how the gradients will flow after that then  ?
Even after doing that i get the same error –
RuntimeError: Need gradOutput of dimension 4 and gradOutput.size[2] == 5 but got gradOutput to be of shape: [20 x 512 x 4 x 4] at /Users/soumith/miniconda2/conda-bld/pytorch_1490984005896/work/torch/lib/THNN/generic/SpatialConvolutionMM.c:51