ptrblck:

Do you know how much memory a single model would take? Based on the description of your use case I guess you are working on an inference use case? My guess would be that 6GB are quite limited to host 4 BERT models, but it of course depends on the particular model architecture you are using.


I don’t know the memory usage of every model, but yes I need the 4 models to detect and infer emotions on texts (I use two “emotion classifier” models, one “sentimment analisys” model and one “toxocity classifier” model. May be 6 Gb is quite limmited.



 ptrblck:

Is this process loading other models as well, as it’s using GPU memory, too?


No this process is the only one that loads the models, make predictions using the models and returns the result using JSON



 ptrblck:

All PyTorch objects are created on the CPU by default. Unless you push them manually to the GPU via .cuda() or to('cuda') the GPU won’t be used.


Exactly,… I load the models and I move to the GPU using “model.to(device)” where device is a var that if is there a GPU stores ‘cuda:0’ value and ‘cpu’ in other situation.
But,… I assume that moving the models to the GPU the python ptrrocess is also moved to de GPU…