TimoKer:

However, I still do not see how the diagram shows the inputs going through a linear/Dense and only then being split. In that case, I would expect only three linear/Dense layers on the figure, not nine.


 I can see what you mean. Notice that the original diagram in the Attention Is All Your Need 1 paper is the same :




 TimoKer:

I am implementing a encoder-decoder transformer for language translation


This full Transformer architecture (in one place) from a popular blog post 4 may be useful, in case you haven’t checked it out. Also, this: Google AI Blog: Reformer: The Efficient Transformer 2.