I’m not sure I understand properly all the points, but here are some tips:

try using index_copy_ instead of advanced indexing (because you used two index_select, which is equivalent to input[:, :, :, idx_w.long()][:, :, idx_h.long(), :]

don’t forget to call .long() in your indexing variables