Hi,
Thank you for your detailed explanation!  I have some other question about changing size of each layer. I hope you can help me.
My purpose is to reduce layer number and expand layer size to maintain the same model size.


When talking about changing layer size, does that mean change hidden_size? For example [768,768] —> [1024,1024].


I tried expanding the tensor size using view(), expand().etc during training. But they result in a dis-matching size with pre-trained model error.  So I assume I have to do pre-train again if I change the layer size?  Reducing layer number does not have this kind of problem.


Best,
ZLK