I use  pretrained Densenet121 to test several datasets, but after running several epochs, the program hangs on and there isn’t any error message reported. The process can be found in host and the memory of GPU is still occupied by the process but the Volatile GPU-Util is 0%
as the follow shows:

image.png860×76 7.67 KB

what’s the problem? Can anyone help to shed  light on this ?
The code:
    for batch_no in range(9,24):
        test_data = datasetMutiIndx(raw_test_data, random_samples[start:start + batch_size])
        complete_data = exclude_wrong_labeled(model, test_data, device)
        test_data_laoder = DataLoader(dataset=complete_data, batch_size=1, shuffle=True)
        deepfool = DeepFool(target_model=model, num_classes=num_out, overshoot=overshoot, max_iter=max_iter,
                            device=device)
        for data, label in test_data_laoder:
            data = data.squeeze(0)
            data, label = data.to(device), label.to(device)
            adv_img, normal_label, adv_label = deepfool.do_craft(data)