You can find an implementation of Attention in PyTorch-NLP 67. I also have implemented the attention mechanism for the hierarchical attention network based on Yang et al. 2016: https://gist.github.com/notnami/a3b4e2e7ff127121c937020c5cd87ace 37. I don’t know if my implementation is the most efficient, but it has worked well in my experiments.