ptrblck:

torch.set_flush_denormal(True)


It did become a tiny bit faster but there is still a lag big between the two models. I manually checked the weighhts and it seems that the weights of both are only till 4 decimal places… Any other suggestions?