First of all, it is very interesting to apply layers with layer scaling. I tried to understand how the deep learning model works with it, but I could not figure out it.
I have some opinion about your questions;
1- Computional graph of your model in the following pic. It is shown how appliying gradinets.
backw405×673 22.3 KB
2- I have no idea.
3- Did you use L2 regularization with SGD? AdamW applies built-in L2 regularization.