Hi Oat!



 oat:

            l2_norm = sum(p.pow(2.0).sum()
                          for p in model.parameters())  # <1>



The first (scanning left to right, rather than in order of operation) sum
is a python sum where you sum over the elements generated by the
“generator” (the expression inside of sum's parentheses that looks like
a python “list comprehension” without the list).  These elements happen
to be pytorch tensors that contain single values.
The second sum, p.pow(2.0).sum(), is a pytorch sum that sums over
the elements of a pytorch tensor (and returns a tensor that contains a
single value).
Just for fun, you can turn the nested sum inside out, first performing a
python sum over the generator (that generates python tensors containing
multiple values), and then performing the pytorch sum that sums over
elements of the resulting tensor (to get a single value):
l2_norm = sum (p.pow(2.0)  for p in model.parameters()).sum()

Best.
K. Frank