I thought the automatic garbage collection should clean those up. For instance, in the above example once the output and loss are recalculated, their old values and any residues kept from the calculation ought to be freed.
But I can imagine that this cleaning up is not as quick as it could be.