Hello,
I have a custom Pytorch optimizer for which I would like to compare the average execution time (with SGD/Adam) during each call to optimizer.step() and also for a specific operation (say matrix-matrix multiplication) within my custom optimizer.
What is the best and correct way to do this without the impact of other background processes?
I want to do this while the network trains to see scalability performance of optimizers, where number of trainable network weights (among all initialized weights) increases over epochs.
This needs be done both for CPU and GPU training.
Any suggestions to do a fair comparison would help. Thanks