@eqy thanks for your suggestion and it sounds great to use resnet , i tried to find its implementation , i have deep fully connected layer not CNN , lot of its use case i found using Resnet in CNN but not in fully connected layer (i assume mine is fully connected layer). Could you please share some naive implementation using fully connected layer?
As you said Transfomers is it type of Resnet as well?
Another question  the embedding size of the intermediate layers is not monotonically decreasing in MLPs what does it mean please?
Thanks