Solved by albanD in post #2 
              
              
                If you use FloatTensor, 1e-8 is close to the smallest number you can represent precisely. At this point most the gradient computation are really noisy. You will need DoubleTensor if you want more precision.