Hi there,
I just got my new RTX 3090 and spent the whole weekend on compiling PyTorch 1.8 (latest master) with the latest CUDA 11.1 (that should support the new 30 series properly).
I did some performance comparisons against a 2080 TI for token classification and question answering and want to share the results 
For token classification I just measured the iterations per second for fine-tuning multi-lingual BERT on GermEval 2014 dataset. With the 2080 TI, I achieved 4 iterations per second, with new RTX 2090 6.8 iterations per second. Using fp16 the 2080 TI achieved 7.7 iterations per second and 8.4 iterations per second with the RTX 3090 (and “native” support of half precision).
I also measured the time that is needed to fine-tune a model on SQuAD. With the 2080 TI it took 38:57 minutes for one epoch and 22:25 minutes on the RTX 3090.
I’m planing to do more benchmarks with other examples. I don’t have any TITAN RTX card, but if someone from the community could provide some benchmarks (also for other cards!!), it would be awesome