Great analysis @KFrank!
@ptrblck  and I were chatting about this and we found that it appears to be solved in the nightlies (@ptrblck found the probable cause: Fix: Compare input against beta * threshold in softplus backwards by jbschlosser · Pull Request #56484 · pytorch/pytorch · GitHub 2 fixing `Softplus` forward and backward discrepancy · Issue #55587 · pytorch/pytorch · GitHub).
Best regards
Thomas