Found the issue and a fix or two:
The issue seems to be caused by patchelf, which alters the order of sections in a way that ldconfig and libc libraries for do not deal with prior to libc version 2.31.
See this issue, https://sourceware.org/bugzilla/show_bug.cgi?id=25087 54.  You can confirm that this is your issue by looking at the section order with readelf -S
I’m currently on glibc 2.30, and upgrading would be a big task and risk bricking my Linux install.  So instead I built libtorch from source.  The compilation at least on my system does not appear to use patchelf, so building it myself fixes the issue.   Followed the instruction in the release notes and it went easily.