index_select would work, but it serves the different purpose.

You will get a tensor of dynamic shape (?xCxHxW) instead of (NxCxHxW)
Backprop through integer index doesn’t work.

I had the same question before, and imho, broadcasting is the best way. Masking is very common in deep learning and people do this way. Maybe it can be improved but not worth the effort.