Not familiar with Horovod implementation. But if it internally uses PyTorch ProcessGroup or DistributedDataParallel, it would work with NVLink, if you specify the nccl backend 3 when calling init_process_group 2.