jafioti:

xa


First, I would suggest to start a model with fewer parameters to prove the idea. It saves time to see the preliminary results before wasting time to train a complex model.
Second, IMO, it would be great to start with the transformer/MultiheadAttention modules in Pytorch library since they are developed and maintained by many people. This should avoid some mistakes during some proof tests.