your method needs to first compute all layers’ representations (or intermediate outputs); and then uses those related outputs to obtain cost for each particular layer; finally, computes the gradients of each layer parameters and updates them respectively.

yeah, but you don’t need to compute all the layers’ representations upfront, this was just an example. You could, for example, fetch the output of the 1st layer, do a gradient update, fetch the output of the 2nd layer, do a gradient update, and so forth.