I am trying to do the following:

Pass an image to a neural network and predict a patch and it’s location
Apply the patch on the image (Change the intensities of image by values of patch at the predicted coordinates)
Pass this modified image to a pre-trained classifier model which is frozen (model.eval())
I want to basically create a patch on an image at some location which would cause adversarial attack.

Below are the chunks of code which I am using for above steps:
class PatchModel(nn.Module):
    def __init__(self, sz):
        super().__init__()
        self.sz = sz
        layers = [Flatten(), nn.Linear(784, 100), nn.ReLU(inplace=True), nn.Linear(100, sz * sz + 2)]
        self.m = nn.Sequential(*layers)
    def forward(self, x):
        img_sz = x.size()[2:]
        o = self.m(x)
        o[:, :2] = torch.clamp(o[:, :2], 0, img_sz[0] - self.sz) 
        return o

This is the model that takes in image and predicts patch and it’s coordinates
def apply_to_img(img, op):
    patch = op[:, 2:].view(1, 1, 4, 4)
    coord = op[:, :2].int()
    _temp = torch.zeros(*img.size())
    _temp[0, 0, coord[0, 0]:coord[0, 0] + 4, coord[0, 1]:coord[0, 1] + 4] = patch
    # Adding patch to image
    img = img + _temp.clone()
    return img

I defined an function which takes image and output of the first neural network and adds the patch to the image.
def epoch(loader, model, opt=None):
    """Standard training/evaluation epoch over the dataset"""
    total_loss, total_err = 0.,0.
    for X,y in loader:
        X,y = X, y
        # Pass the image into Patch model
        yp_ = patch_model(X)
        # Use the apply function to apply patch to image
        yp = apply_to_img(X, yp_)
        # This is frozen pretrained model for which I want to create advesarial patch.
        yp = model(yp)
        loss = -nn.CrossEntropyLoss()(yp,y) + nn.CrossEntropyLoss()(yp, torch.LongTensor([3]))
        if opt:
            opt.zero_grad()
            loss.backward()
            opt.step()
        
        total_err += (yp.max(dim=1)[1] != y).sum().item()
        total_loss += loss.item() * X.shape[0]
    return total_err / len(loader.dataset), total_loss / len(loader.dataset)

This is main training loop
I passes Adam optimizer to the above function:
opt = AdamW(patch_model.parameters(), 0.001)
I am getting the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-150-e28149387ca7> in <module>
----> 1 epoch(train_loader, model.eval(), opt)

<ipython-input-149-05b573ec2ee5> in epoch(loader, model, opt)
     10         if opt:
     11             opt.zero_grad()
---> 12             loss.backward()
     13             opt.step()
     14 

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    100                 products. Defaults to ``False``.
    101         """
--> 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    103 
    104     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91 
     92 

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

What can I do to prevent this error? Also, the error goes away if in the apply_to_img function I detach the output of my PatchModel network but then, as you would expect, the PatchModel parameters don’t get updated. It remains same after running epoch.
I am using MNIST data with batch size 1 hence I directly hardcoded the indices in apply_to_img function