So I ran this by people who know more than me (i.e. everyone else), and @ngimel kindly explained that this is the expected behaviour for tf32 arithmetic, which only uses 10 bits of mantissa.
There is an issue that discusses the loss of precision here: The matrix multiplication operator can't get correct results on 3090 !! · Issue #61890 · pytorch/pytorch · GitHub 8
Best regards
Thomas