Here is how I implement my custom L2. Can anybody verify if it is correct ?
Here is getParameters function, which take all parameters of sub-model and flat it so I can get norm easily
def getParameters(self):
    """
    Get flatParameters
    note that getParameters and parameters is not equal in this case
    getParameters do not get parameters of output module
    :return: 1d tensor
    """
    params = []
    for m in [self.ix, self.ih, self.fx, self.fh, self.ox, self.oh, self.ux, self.uh]:
        # we do not get param of output module
        l = list(m.parameters())
        params.extend(l)

    one_dim = [p.view(p.numel()) for p in params]
    params = F.torch.cat(one_dim)
    return params

I add my custom L2 to err before I call backward, then step
Only err is Variable. err is output of criterion(output, target).
But l2_model and l2_emb_params,  batch_size are not Variable (they are float and int)
    params = self.model.getParameters()
    params_norm = params.data.norm()
    l2_model = 0.5*self.args.reg*params_norm*params_norm
    emb_params = list(self.embedding_model.parameters())[0]
    emb_params_norm = (emb_params.data - self.emb_params_init).norm()
    l2_emb_params = 0.5 * self.args.embreg* emb_params_norm * emb_params_norm
    err = (err + l2_model + l2_emb_params) / batch_size

    err.backward()

    # after loop batch_size
    optim.step()
    optim.zero_grad()