Here is the solution I came up with. It is ugly but does not need a loop:
    nLastDim = t2.shape[-1]
    nLast2Dim = t2.shape[-2]
    nLast3Dim = t2.shape[-3]
    lastDimCounter = torch.arange(0,nLastDim,dtype=torch.long)
    last3DimCounter = torch.arange(0,nLast3Dim,dtype=torch.long)
    t2 = t2.reshape(-1)[(idx*nLastDim+(last3DimCounter*nLastDim*nLast2Dim).unsqueeze(-1)).unsqueeze(-1).expand(-1,-1,nLastDim) + lastDimCounter]