Here is the snippet:
    for ii,line in enumerate(lines):

        if line['evidence'][-1][-1][-1]  != None:
            #feats, labels = tf_idf_claim(line)
            feats, ev_sent, _ = fasttext_claim(line,ft_model,db,params)
            labels = ind2indicator(ev_sent,feats.shape[-1])
            all_labels.append(labels.numpy())

            pred_labels, scores = model(feats.unsqueeze_(0).transpose_(1,2).cuda(device_id))
            all_scores[ii] = scores

I see that the gpu errors out OOM after a while. This is running on Tesla K80.
If I change the last line to all_scores[ii] = scores.detach().cpu().numpy(), it gets fixed.
This is only process running on the gpu core.