I found another example of someone trying to use nn.TransformerEncoder for sequences classification 58 – unfortunately their model doesn’t seem to be learning anything either, accuracy on IMDB is 53% on the training set. I contacted the author who thinks it may be attributable to suboptimal hyperparameters, I but I’m not so sure. I’ve tried with similar hyperparameters to what the former example above uses with no improvement.
Speaking of which, I was reading through @pbloem’s excellent article on transformers 23 and realized that he was the author of former, which I’ve been using as an example for a few weeks, and also realized that he seems to be registered on this forum. If you happen to see this post and can spare a minute, and if you can immediately see an obvious reason why nn.TransformerEncoder is failing at classification where your former library does great, I’d be truly grateful if you could point me in the right direction.
EDIT: Also tried asking at r/learnmachinelearning 10, figured I might as well leave a link in case that gets a response that could help someone.