We had a similar problem when running inference, with some changes to libtorch that led to an unexpected P2P memory access from GPU0 to GPU1. If you disable IOMMU in the kernel setting the hang goes away. We are on 440.59. The problem started after we upgraded our driver. You would also see things like DMAR:[fault reason 05] PTE Write access is not set DMA write in the kernel log.