optimizer_state["found_inf_per_device"] is populated in the unscaling step, which is either called explicitly or inside the scaler.step(optimizer) call. How are you using amp and which approach did you pick?