Thank you for you reply! So about the first issue, when I train the NN, the output of the network becomes a part of the loss (I compare the output pixels with the real ones). But when I backprop the loss, the derivative of each pixel in the image will have its derivative taken with respect to the previous layers anyway (which is what I need here). My point is, aren’t the derivatives that I need here also computed during the actual NN training (the only difference is that during the training I calculate the mean of the pixels in order to get just one number, but there are still 3x64x64 weights). For example assuming that I have the chain a->b->c->d->loss, and I want the derivative of d with respect to b, isn’t that derivative also computed when I take the derivative of the loss with respect to b? Is this wrong?
For the second issue, you are right, it says 4.94 MiB free, but where is the rest? The math doesn’t add up?