Nate:

the correct statement for the gradient computation involved in the regularization term? At first glance, it appears to be allowing for the computation of the gradient of the difference of the losses, rather than the difference of the gradients. But, I may be misinterpreting this line completely.


The gradient operator ∇ is in itself linear as an operator. This means that ∇a - ∇b = ∇(a-b).
In fact, if you write a minus function m(x1, x2) = x1 - x2 and some loss l(m) calculated from that, then the backpropagation step for this calculation is just feeding dl/dm as grad_out into the backward of x1 and -dl/dm as grad_out into the backward of x2. So really, the two not only agree mathematically but also computationally, with some efficiency gain if the two have common intermediates, so that the graph of the calculation of the intermediates has to be only backpropagated through once.
Best regards
Thomas