This parts is just additional regularization + sample weights:
if use_weights:
weights = batch[4].to(self._device)
loss = (loss * weights).mean()
if l1_weight:
weights_norm_1 = 0
for i, params in enumerate(self._model.parameters(), 0):
if i != 0:  # not embedding layer
weights_norm_1 += torch.norm(params, 1)
loss = loss + l1_weight * weights_norm_1
Where can I use .grad_fn to understand if they have childs?
List item