You could directly index the tensor without using gather.
I’ve changed the shape a bit (used stack instead of cat and created a final output tensor) for a better comparison:
index = torch.tensor([[0, 0, 0, 2], [0, 0, 1, 2]])
input = torch.randn(2, 3, 2, 3)

output0 = torch.stack( (input[0, index[0,0], :, :], input[0, index[0,1], :, :], input[0, index[0,2], :, :], input[0, index[0,3], :, :]))
output1 = torch.stack( (input[1, index[1,0], :, :], input[1, index[1,1], :, :], input[1, index[1,2], :, :], input[1, index[1,3], :, :]))

output = torch.stack((output0, output1))

res = input[torch.arange(2).unsqueeze(1), index]
print((res == output).all())
> tensor(True)