Pinterest, Engineering, Monetization.
cross layer, was sometimes overflowing during training with float16. This might be mitigated by tuning some hyperparameters (e.g. weight decay), but a slight risk would still remain, and a failure mode of “complete failure, no score predicted” is not ideal. The main pitfall of bfloat16 is that due