Sorry for the late response.
First, mixed precision speedup will only work on Volta (Titan V, GV100) and Turing (RTX 20xx) Architectures. Your GTX 1050 will not benefit from this and will still use FP32 computation underneath.
Second, mixed precision can affect your performance. However, based on results I’ve seen, the performance difference will be minimal when trained right.
If your performance is bad then it could be due to several reasons, and I’d try to converge using FP32 first, unless you have a very strong reason to believe that your existing code must converge.