I was trying to implement several runs for one model to see the confidence interval of my model prediction. However, it becomes very slow.  The following are the sample codes.
    for i in range(opt.ite + 1):
        model = getattr(models, opt.model)()
        if opt.load_model_path:
            model.load(opt.load_model_path)
        model.to(device)

        model.apply(weight_init)

        # optimizer
        op = Optim(model.parameters(), opt)
        optimizer = op._makeOptimizer()

        previous_err = torch.tensor(10000)
        best_epoch = 0
  
        for epoch in range(opt.max_epoch + 1):
          
            # load the data
            for data in zip(datI_loader, datB_loader):
                # train part                   
                loss.backward()
                optimizer.step()
       
            if epoch % 100 == 0:
                test_err = val(model, grid, sol)
                if test_err < previous_err:
                    previous_err = test_err
                    best_epoch = epoch
        test_meter.add(previous_err.to('cpu'))
        epoch_meter.add(best_epoch)

For example, if I take opt.ite to be like 2 or 3, it will becomes extremely slow, which might take 5 or 6 hours, however, I remove the outer for loop, which, looks below,
        model = getattr(models, opt.model)()
        if opt.load_model_path:
            model.load(opt.load_model_path)
        model.to(device)

        model.apply(weight_init)

        # optimizer
        op = Optim(model.parameters(), opt)
        optimizer = op._makeOptimizer()

        previous_err = torch.tensor(10000)
        best_epoch = 0
  
        for epoch in range(opt.max_epoch + 1):
          
            # load the data
            for data in zip(datI_loader, datB_loader):
                # train part                   
                loss.backward()
                optimizer.step()
       
            if epoch % 100 == 0:
                test_err = val(model, grid, sol)
                if test_err < previous_err:
                    previous_err = test_err
                    best_epoch = epoch

It only take about 20 mins to finish.
Any ideas about this problem?