If I understand correctly, sometimes you are trying to backpropagate with respect to a mask (an input?) and sometimes with respect to the parameters, but not both at the same time?
Instead of deleting them, you can just set requires_grad=False when trying to get the mask gradient. Something like:
for param in model.parameters():
   param.requires_grad = False

mask = ... # mask should require grad
loss = model(input1, mask)
loss.backward()  # get gradient for mask

...

for param in model.parameters():
   param.requires_grad = True
loss = ...
loss.backward()  # get gradient for parameters