Progress so far
My current solution is to include a second loss function:
class CoefficientLoss(nn.Module):
    def __init__(self, alpha):
        assert isinstance(alpha, torch.Tensor), 'Whoops, alpha is not a Tensor.'
        self.alpha = alpha
    def forward(self, weights):
        return self.alpha * (1 - weights.norm())**2

Then one simply calls
loss = criterion(output, input) + criterion_aux(criterion.weights)

where weights are no longer normalized using F.normalize (because this new loss function is serving that purpose), and where
criterion = WeightedLoss(n_weights)
criterion_aux = CoefficientLoss(torch.Tensor([1000.]))

This gives me a close approximation to what I need (but not precisely what I want!).
Feasible direction
No pun intended.
It turns out that this code 4 (linked in OP) actually provides a lovely example of defining a backwards method manually, but it took me a bit of staring to parse what was going on. It’s still a little fuzzy. In short, it should be possible to define a WeightedLoss using a WeightedLossFunc that has a backward method that returns the appropriate gradients. For my particular use case this seems rather involved — that is, I’m still unsure how to proceed with this avenue, and any suggestions to this end would be greatly appreciated.