Solved by tom in post #4 
              
              
                I think that that is just a simplification in the implementation. The blue box note below the AttnDecoderRNN alludes to the fact that this approach is more limited than others. 
What happens with the overly long attention weights is that they will get multiplied by the 0s in decoder_outputs. 
Having…