So, finally how can we solve this problem?
I use pytorch to accurate the large matrix operation rather than training deep net. However, the precision garbage brings the propagated error, which make the final result fail.