It wasn’t for transformers, but ULMFiT. Didn’t get the chance to try it on transformers model.
Also, I was using the manifold mixup 9 version, which applies the mixup at a random layer (not necessarily the embedding), though this could also mess up the attention mechanism in tansformers.