I think you are looking for torch.gather 5, with features os size (32, 128, 40, 40) and idx (the permutation for each feature map in the batch) os size (32, 128), this should do the trick:
permuted = torch.gather(features, idx, dim=1)