Solved by David_Reiss in post #4 
              
              
                This is a bug.  Fix is https://github.com/pytorch/pytorch/pull/31582 .  Should be available in the next nightly build.  Sorry for the trouble.