I don’t really know how to check that.
My worry that the following is happening. I do an iteration where the loss is over 1 and I don’t do a backwards on the loss, so all tensors still hold their gradient trees from the iteration. Then, on the second iteration the gradient trees are then also changed according to the information of the new iteration (while still holding the information of the first iteration) and if the second iteration loss is then lower than 1, the gradients are calculated, but there is also the information about the first iteration that was actually lower than 1.