Well… this is embracing, but unfortunately the TrannsformerEncoderLayer documentation doesn’t seem to contain the information (rather the Transformer class does) that the input is (sequence length, batch, embedding dim), instead, I pushed (batch, sequence length, embedding dim).
The reason why it trained at all was that I would either select the CLS embedding that was 768 dimensional or averaged across the dimension so that I get the 768, which was required for the classification network. And the miracle that average would actually start training when I leaked the label… well, neural networks work in a mysterious ways.
In any case, any additional advice on general sequence modeling is welcome.