Yeah, well, the core of the problem is that you are not using scores or val in the further calculation. If you then do the calculation based on the input sequence, you cannot expect gradients (which factor through dloss/dval and dloss/dscores = dval/dscores dloss/dval) to flow back. (You need to use scores.retain_grad() to keep intermediate gradients, btw.)
Now, if you consider only the indices as the output of your selection model, you could look towards learning techniques in reinforcement learning for discrete decisions.
Best regards
Thomas