I have find a better approach @bonzogondo :

  
    
    
    API Rest with several models loaded using GPU but not at same time 
  
  
    Thank you for comming to this issue  
I have tried that approach, the problem that I have seen is that if I don’t delete pred variable after using, it keeps memory in use and when I load other model cuda memory error is thrown!
  


models = [...] # define a list of all models on the CPU

input = ... # get your input
for model in models:
    model.to('cuda')
    pred = model(input)
    #make something with pred
    del pred
    model.to('cpu')

They key fact is that you need to delete predictions variable if you don’t wank cuda memory issues