torch.cuda.set_device(hvd.local_rank()) is how I set device for each rank. hvd.local_rank() is printing the correct GPU number. So Horovod is not to blame here.
What else should I check? What are the cases where moving the model to the GPU would cause "illegal memory access"?