If I am running a conv net or a standard architecture like AlexNet, how does one collect stats of the model during training in an efficient way? Like the norm weights, norm gradients, test, train errors, etc.
Is it just something like:
train_errors.append(loss.data.numpy())

inside the for loop? (and for the params stats just looping through all layers and stuff). I was just worried that doing that too often (brining data to cpu) could be inefficient so wanted to see what the good practices were and make sure I didn’t do anything silly.
I was also just wondering if there was anything in particular that I had to be careful, perhaps only using a mini-batch or something like or only recording the error & loss as a running average during each epoch (so we run the model over the data set once and at the end an average of the loss & error of the model for each mini-batch so running_average_for_one_epoch = sum_{mini batches} loss_per_minibatch / |total number of mini batches used|).
Is there anything in particular I should be careful about?