There is a tf way of doing this used here https://github.com/tianheyu927/PCGrad/blob/c5fbd7c856526373828074f06875230f7f3ee79e/PCGrad_tf.py#L39 11. Isn’t something like this possible in Pytorch?
If not ,then are there anyother ways to optimize the above code my gpu still runs at around 50% usage and increasing the batch size is deterimental to my results?