Hi,
The operation that goes from your first network output to the index is non differentiable (or it’s gradient is 0 almost everywhere). So you cannot backprop through that operation.
A possible solution could be to do a soft indexing on your embedding layer based on the continuous output of your first network. But that is not supported by Embedding so you will need to do this by hand with a matrix vector product using the weights of the Embedding layer.