I do not know tensorflow very well, so I am not sure if I got what you are asking correctly, can you check if thats what you are looking for:
>>> lookup=torch.ones((15, 720, 30))
>>> children=torch.randint(0,15,(15, 720, 19, 2),dtype=torch.long)
>>> lookup[children[:,:,:,0],children[:,:,:,1],:].shape
torch.Size([15, 720, 19, 30])