I was hoping to replace one Dense Layer with a Transformer in image classification for hopefully better performance. I’m hoping to replace the classifier section after the feature extraction with a transformer block.  Does anyone have any tips/code to show how to do this? My current issue is that most transformer modes have target mask, but I’m guessing that won’t help when replacing a Dense Layer, and may even hurt it. I also think that since we are not using text, embedding is not needed. I’m guessing that positional encoding is useful, given that orders of features matter, but I’m not sure if a standard Positional Encoding to Transformer Block to Linear layer setup would work at all.
Any tips?