I’m sure there is a better way of indexing, but this could work for now:
a, b = batch_mask_agent_position.split(1, dim=2)
ret = action_grid_batch[torch.arange(action_grid_batch.size(0)), :, a, b]
ret = ret[torch.arange(ret.size(0)), :, torch.arange(ret.size(2))]
print(ret)
> tensor([[[0.5000, 0.5000],
           [0.2000, 0.8000],
           [0.9000, 0.1000]],

          [[0.7000, 0.3000],
           [0.4000, 0.6000],
           [0.1000, 0.9000]]])