Well, for sure the code is very optimized to run as it is. To create such a operation you should rewrite backward function which will be slower.
Anyway, if you want to do that you can just use 3D convolutions with kernel=1 padding= 0 and stride =1 (in the 3rd dimension). That would be exactly what you are asking for.