Hi everyone!
I have a project where I have to build, train and analyze a simple transformer for language modeling.
I built my model following the tutorial in the pytorch examples:
https://pytorch.org/tutorials/beginner/transformer_tutorial.html 6
Here is my jupyter notebook: https://github.com/andcarnivorous/TestingTransformers/blob/master/transformertest.ipynb 12
I have tried to implement the input and target masks and so I started training on the wikitext dataset different versions with more/less encoders, decoders and heads. What I have noticed is that even when I use only two heads and two encoders/decoders the model manages to get to quite low validation loss.
Any feedback on how I set this up would be really appreciated.