I think Tensor Comprehensions 38 might be a good fit for your use case.
You can synthesize high-performance kernels and use them in PyTorch.
Have a look at some conv examples 57 to get an idea, what to do.