ptrblck:

How are you defining the multi-GPU use case?


In multi-GPU usecase I start one triton server on two GPUs and place one instance of the torchscript model on each GPU using triton config.



 ptrblck:

Could you explain your deployment


I am using  triton server 2. What other information is required?



 ptrblck:

where it’s currently breaking?



The triton server is running on cuda:0 and cuda:1

Torchscript was complied using cuda:0


The error is caused by model instance on cuda:1  of the server fails with the error message expected device cuda:1 but got device cuda:0