Since the gradient update is usually done at CPU while Forward and backward are done on GPU, you could remove the .cuda() call at these parts of your code



 Gkv:

loss=criterian(pred,target).cuda()





 Gkv:

criterian=nn.MSELoss().cuda()


and put this part of your code before moving the model to GPU:



 Gkv:

optimizer=torch.optim.Adam(model.parameters(), lr=l_rate)


Your run-function would now look like this:
def run():
    torch.cuda.empty_cache()
    n_epochs=100
    print("Running...")
    global n_batches
    n_batches=100
    global model
    model=my_model()
    optimizer=torch.optim.Adam(model.parameters(), lr=l_rate)

    model = model.cuda()

    model=nn.DataParallel(model,device_ids=[0,1,2])
    criterian=nn.MSELoss().cuda()
    train_ids=find_dataset(full_train_set1) # train_ids is a list 
    l_rate=0.001
    
    train_loader = torch.utils.data.DataLoader(dataset=MyCustomDataset(train_ids,train=True),
                                                batch_size=n_batches,
                                                shuffle=True)

    for epoch in range(1,n_epochs+1):
        gc.collect()
        train_loss=train(epoch,train_loader,optimizer,criterian)
        print('Epoch :'+str(epoch)+': Train rmse:',train_loss)


This would prevent loss function and optimizer from living on GPU (and thus decrease the GPU memory usage). However, since you run OOM on CPU, I would first try to load only the parts of your dataset you need just-in-time instead of loading the whole (probably huge) datasets before and cache them. Saying that, your CPU RAM should not increase but stay approx. at the same level of usage.