DeepNet: Scaling Transformers to 1k Layers
A lot of previous transformer-based models did not have great depth simply because "shallow" transformers already took forever to train.<p>I am concerned that we are approaching an age where only massive companies and research groups with tons of GPU resources will be able to train next-gen models.