@ptrblck I am using BCE With Logits because regular BCE is not half-precision supported (I got an error to that effect when switching to half precision).  So I am using BCE w/ Logits and just tagging sigmoid on.  I suspected sigmoid, but I created a float16 tensor and ran it through sigmoid fine on GPU.  Sigmoid on CPU complains about half precision.  I am going to capture every value and find out what exactly is the issue.
I have since moved my label smoothing outside of the clause, so I modify y before the clause. My label smoothing is just basic, I am just modifying y in a basic way, there is nothing more to it than that.  For loss its BCE with Logits Loss.