I wasn’t using optimizer but I cleared the grad for each layer’s weights which solved the problem. I have a doubt if optimization based on the gradient would work or not because I cleared the gradient for the weights.