The module is usually called Locally connected (conv) layer: https://prateekvjoshi.com/2016/04/12/understanding-locally-connected-layers-in-convolutional-neural-networks/ 25
This module is available in Keras: https://keras.io/layers/local/ 11
Original function requested in Pytorch: https://github.com/pytorch/pytorch/issues/499 25
Please take a look at this pull request in Pytorch (reports error): https://github.com/pytorch/pytorch/pull/1583 27
It wraps a function called SpatialConvolutionLocal from torch community. However, it does not work due to weird error.
I tried to replace the original foward() with this:
def forward(self, input):
    return self._backend.SpatialConvolutionLocal.apply(input, self.weight, self.bias,
           self.kernel_size[1], self.kernel_size[0],
           self.stride[1], self.stride[0],
           self.padding[1], self.padding[0],
           self.in_width, self.in_height,
           self.out_width, self.out_height)

It can pass the forward(). But report the following error during the backward().
File “/home/yuhang/Program/MultimediaFastMatching/sphereface_pytorch/test_locallyconnected.py”, line 24, in 
loss.backward()
File “/home/yuhang/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py”, line 167, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
File “/home/yuhang/anaconda2/lib/python2.7/site-packages/torch/autograd/init.py”, line 99, in backward
variables, grad_variables, retain_graph)
File “/home/yuhang/anaconda2/lib/python2.7/site-packages/torch/autograd/function.py”, line 91, in apply
return self._forward_cls.backward(self, *args)
File “/home/yuhang/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/thnn/auto.py”, line 210, in backward
return (backward_cls.apply(input, grad_output, ctx.additional_args, ctx._backend, ctx.buffers, *tensor_params) +
RuntimeError: _Map_base::at