You’re right I just realized that the gradients in the a and e position sum together to the scale gradient. Thanks! But since I’m here I wanted to ask a related question - why is it that when the gradients are largest, the values are affected the least? I’m noticing that when I plot my gradients of x, the larger they are (they oscillate between super big and super small) yet the value of x doesn’t change at all… I’m using an Adam optimizer and was wondering if thats why? Not sure how Adam affects it but just a guess…