Solved by soulitzer in post #4 
              
              
                Setting the filter to not require grad in a convolution is fine (see below): 
import torch
a = torch.rand(10, 5, 5, 5, requires_grad=True)
b = torch.rand(5, 5, 2, 2, requires_grad=False)
out = torch.max(torch.nn.functional.conv2d(a, b)) 
out.requires_grad # True

We’d probably need some more context…