My slightly better solution is:
n_samples, topk = len(x), 3
s_inds = torch.arange(len(x)).long()
top_y = y[s_inds.repeat_interleave(topk),:,top_inds.view(-1)] #(n_samples*topk,C)
top_y = top_y.view(n_samples,topk,-1).permute(0,2,1) #(n_samples,C,topk)