Thanks a lot, I will check your code. Multiplicator is part of my groundtruth, so it should not have a graph. I checked before and am fairly confident that this is not an issue due to accumulating graphs, but should occur within the first call, but I will verify that again. Currently I try to insure that by

del backprop_loss

in every iteration after the optimizer step etc.
The value of n_predictions is my batch size, which used to be 16, but I set it to 10 now and it seems to not run out of memory with that. But again I will confirm that. I cannot currently link my code unfortunately, it may be published in a while though, I will link the git here then.
Thanks a lot for the code, my implementation did in fact slow down everything noticably, which I attribute to the two loops. I will check again with a bigger batch size and see if your code fixes the issue.