Solved by tom in post #2 
              
              
                Yeah, well, the core of the problem is that you are not using scores or val in the further calculation. If you then do the calculation based on the input sequence, you cannot expect gradients (which factor through dloss/dval and dloss/dscores = dval/dscores dloss/dval) to flow back. (You need to use…