I just realized the error states: torch.LongTensor or torch.ByteTensor as the only argument at the very end. So, I guess input[:, perm] will not work.
I can probably do a dimension permute to bring the required dimension to dim0 and do the permutation and dimension permute back to original order.
Seems like a hacky way to do it though.