Hello Elidor!



 Elidor:

What I would like to obtain as the output of this forward function is a tensor with shape ([50, 61, 800]) instead of two tensors made in that way.
Do you think there is a way to do it without “affecting” the attention calculation?


I don’t have any advice for you, as I don’t have any intuition about what
these values might mean.
Perhaps if you could give a very high level description of your use case,
some expert in that area might have suggestions.  Are you working on
some sort of (semi-) standard model?  Is this a research direction that
others might have experience with?
Let me also reiterate that, at a lower level, a core issue is the amount
of “information” you have.  If you want to turn [50, 61] and [50, 800]
into [50, 61, 800], you either have to get the additional information
from somewhere else, or your [50, 61, 800] has to encode the limited
information you have in some redundant way (such as using  the outer
product I mentioned earlier).
Good luck.
K. Frank