Jeronimo_Molina:

I’m loading 4 (“only four…”) BERT models… yes the four models are really large… I’m working on Emotive Computing.


Do you know how much memory a single model would take? Based on the description of your use case I guess you are working on an inference use case? My guess would be that 6GB are quite limited to host 4 BERT models, but it of course depends on the particular model architecture you are using.



 Jeronimo_Molina:

the last process is a flask web service that loads the models and make predictions.


Is this process loading other models as well, as it’s using GPU memory, too?



 Jeronimo_Molina:

Can I move python and the WS to the CPU memory (I think it is not possible…)?


All PyTorch objects are created on the CPU by default. Unless you push them manually to the GPU via .cuda() or to('cuda') the GPU won’t be used.