Thanks for the code update.
Could you explain, why the first output should be different, if you pass in the same tensor?
If I’m not mistaken the submodules are mutliheadattention as well as some linear layers and I’m not sure, why they should yield a different output. However, not that I haven’t looked into the implementation deeply.