The difference between pytorch and lua torch is probably due to autograd overhead (it is ~10us per operation in pytorch). There are plans to reduce autograd overhead, this should help bring pytorch time closer to lua torch (see https://github.com/pytorch/pytorch/issues/2518#issuecomment-327835296 103. As for speeding up pointwise operations, fuser to do this in on the roadmap. In the meantime the best solution is probably to have your custom kernels using cupy, like in pyinn project https://github.com/szagoruyko/pyinn/blob/master/pyinn/conv2d_depthwise.py 141. You’d also need to hand-code your backward pass in this case.