So you manually compute the forward mode gradients in Z?
But if you compute the forward mode gradient of the model that gives you the loss, then Z should be the same size as loss which is just 1 element.