I’m not sure if I misunderstand the use case, but wouldn’t this matmul work?
N, F, C, W = 2, 3, 4, 5
x = torch.randn(N, F, C, W, W)
y = torch.matmul(x.unsqueeze(2), x.unsqueeze(1))
print(y.shape)
> torch.Size([2, 3, 3, 4, 5, 5])