How can a switch to different data type create such a large error? It is just a simple dot product. If this kind of dot product can create such a large error with fp16, then it will should not be possible to train a neural network using fp16.
Doesn’t this indicate that there is something buggy here?