I don’t know if it would solve your problem, but you can use gather with different tensor sizes by repeating tensors along added dimensions:
values = torch.rand(3,7)
indices = (7*torch.rand(5,6)).long() # indices for values's 2nd dimension
repeated_values = values.view(3,7,1,1).expand(3,7,5,6)
repeated_indices = indices.view(1,1,5,6).expand(3,1,5,6) # note the 1 on the dimension for gathering
gathered = repeated_values.gather(1, repeated_indices).squeeze() # tensor of size 3,5,6