I guess this line of code:
l2[:, pn] = 0.05 * l2[:, pn] - point_cls[b, pn, 1]

might be causing the issue, as you are assigning the new values inplace to l2.
As a workaround you could append the results to a new list and create the tensor afterwards.