The good effects of half precision floating points may just be negligible in this case
U said u were using mixed precision right? Then just making it a single precision might give u what u want I guess