Are you by any chance confusing permute with index select?
http://pytorch.org/docs/master/tensors.html#torch.Tensor.permute 11

permute(*dims)
Permute the dimensions of this tensor.

Parameters:	*dims (int…) – The desired ordering of dimensions


#Example
>>> x = torch.randn(2, 3, 5)
>>> x.size()
torch.Size([2, 3, 5])
>>> x.permute(2, 0, 1).size()
torch.Size([5, 2, 3])