Then, why don’t you stack your two outputs? Then you can treate them as a single tensor, and you just divide them inside the backward in order to treate them differently:
class MyFunction(Function):
    def forward(self, input):
        #...
        return torch.stack([output1,output2], 0)

    def backward(self, grad_output):
        grad_output1, grad_output2 = torch.split(grad_output, 1, 0)
        #...




 Nadav_Bhonker:

By the way, the context is nearest-embed layer for the VQ-VAE model.


Wow, I didn’t know that paper, the results with videos are amazing  I’m looking forward to seeing your implementation working as well!
By the way, now I’m thinking about this idea of nearest embedded applied to reinforcement learning. If someday I have available time