I don’t use set_devices, just copy model to gpu when creating the instance and copy input data to a specific gpu at each inference.
I am running this script on gpu:0 and gpu:1, and I do run other processes on gpu:2 with other scripts. I think they should be running independently.
The error is still there without using amp.
It is a flask service, and it receives and process the requests simultaneously. The models are created in the main thread, but flask processes requests in multiple threads, each threads get one model instances(put in a queue) to do infererence.