DeepNet: Scaling Transformers to 1k Layers
A lot of previous transformer-based models did not have great depth simply because "shallow" transformers already took forever to train.I am concerned that we are approaching an age where only massive companies and research groups with tons of GPU resources will be able to train next-gen models.