@Tony_Gracious in my case, it was because I was initially train the model using nn.DataParallel with one GPU, then once I reload the model, it seems DataParallel still store the previous device_ids, hence the single GPU. Now I manage to solve it by every time I load the model, I will re-wrap my model with nn.DataParallel
model = _load_model()
model = nn.DataParallel(model.module)

You also need to be careful for your data size. I need to modify model = nn.DataParallel(model.module) into model = nn.DataParallel(model.module, dim=1) since I am using batch_first=False